Teman DevOps note: 2015

2015年12月9日星期三

Read Note - Grails Starter.

Grails Doc https://grails.github.io/grails-doc/latest/guide/single.html#coreFeatures , easy make controll,view and dmodel together, come with groovy and spring , and use gradle build system

2015年12月3日星期四

Next Generation Eclipse ?

Che Eclipse : http://www.eclipse.org/che/

it's web based (browser) good theme , but no shortcut and assistant for me to ready use...

How to use maven launch embedded tomcat or jetty with plugin

Tomcat maven plugin

currently support tomcat7
http://tomcat.apache.org/maven-plugin-2.2/context-goals.html

setup pom.xml


    org.apache.tomcat.maven
    tomcat7-maven-plugin
    2.2

$> mvn tomcat7:start

jetty maven plugin

http://www.eclipse.org/jetty/documentation/current/jetty-maven-plugin.html
resources in ${project.basedir}/src/main/webapp
classes in ${project.build.outputDirectory}
web.xml in ${project.basedir}/src/main/webapp/WEB-INF/

    
    org.eclipse.jetty
    jetty-maven-plugin
    9.3.1-SNAPSHOT
    
        10
        
            /test

$>mvn jetty:run

2015-12-3 digest

Maven

swagger maven pluginl swagger maven plugin for generate restful api document

AWS

Version 2 of the AWS Mobile SDK aws network connections for mobile
Authenticating Requests in Browser-Based Uploads Using POST let custom user upload content directly to Amazon S3

BigData

pokemon or bigdata guess!

Github

http://blog.winwu.today/2013/06/githubio.html how to setup github io page

2015年12月1日星期二

MongoDB Clustor

Router (mongos) X2

sudo -u mongo /data/mongodb-3.0.0/bin/mongos --port 27017 --configdb CNF01:27017,CNF02:27017,CNF03.SJC1:27017 \ 
--logpath /data/logs/ROUTER.log --logappend --fork

Config DB Server X 3

/data/mongodb-3.0.0/bin/mongod --port 27017 --configsvr --dbpath /data/dbs/CNF \ 
--logpath /data/logs/CNF.log --logappend --fork

SRD X3 , ReplicaSet X3

ulimit -n 65535
sudo -u mongo  numactl --interleave=all /data/mongodb-3.0.0/bin/mongod --port 27001 --replSet ReplicaSet_01 --dbpath /data/dbs/SRD1  \
--logpath /data/logs/SRD1.log --logappend --fork

Start with config file

$> mongod –f mongodb.conf

Setup

ReplicaSet

rs.initiate()
rs.conf()
rs.add("SRD2:27017")
rs.addArb("ARB1:27017")

Shards

mongo> sh.addShard("ReplicaSet_01/SRD1:27017") # auto add rest replica

Enable shard

mongo> sh.enableSharding("Db")
sh.shardCollection("Db.collection", key: {"key":1})

2015-12-1 Read Note

10 Awesome Docker Tutorials to Kick-Start your DevOps Projects
Next Generation Eclipse IDE : Che naming by me ?XD I would try it with docker container tomorrow

2015年11月30日星期一

AWS Opswork for windows , customize layer / recipes

AWS Opswork stack ,

if you want to use windows , custom layer only support windows 2012

required cookbook

aws Cookbook https://github.com/chef-cookbooks/aws
Windows Cookbook https://github.com/chef-cookbooks/windows
java https://github.com/agileorbit-cookbooks/java

write your recipe

install java , setting custom json and set receipe to custom layer, "java"

{
  "java" : {
      "install_flavor": "windows", 
      "package_name":"Java SE Development Kit 8 Update 20 (64-bit)",
      "url" : "s3://backet/XXXX"
  }
}

install awscli for windows (I don't like powershell) default.rb

if kernel['machine'] =~ /x86_64/
  default['awscli']['url']          = "https://s3.amazonaws.com/aws-cli/AWSCLI64.msi"
else
  default['awscli']['url']          = "https://s3.amazonaws.com/aws-cli/AWSCLI32.msi"  
end

default['awscli']['checksum'] = nil
default['awscli']['package_name'] = "AWS Command Line Interface"

your_recipe.rb

pkg_checksum = node['awscli']['checksum']

windows_package node['awscli']['package_name'] do
  source node['awscli']['url']  
  checksum pkg_checksum if pkg_checksum 
  action :install
end

install .net3.5 feature for windows, default windows 2012 only .net4

Chef::Log.info("****** install .Net3.5 feature  ******")
windows_feature "NetFx3" do
  action :install
  all true
end

install tomcat

pkg_checksum = node['tomcat']['checksum']

windows_package node['tomcat']['package_name'] do 
  source node['tomcat']['url']
  checksum pkg_checksum if pkg_checksum 
  action :install
end

windows_service node['piston']['tomcat']['service_name'] do
  action :configure_startup
  startup_type :automatic
end

Chef::Log.info("******Add firewall rule******")
#TO-DO: need to modify port as parameter
batch "Add firewall rule for Tomcat Server" do
  code <<-eoh data-blogger-escaped-action="allow" data-blogger-escaped-add="" data-blogger-escaped-advfirewall="" data-blogger-escaped-end="" data-blogger-escaped-eoh="" data-blogger-escaped-firewall="" data-blogger-escaped-localport="8080" data-blogger-escaped-name="Open Port 8080" data-blogger-escaped-netsh="" data-blogger-escaped-pre="" data-blogger-escaped-protocol="TCP" data-blogger-escaped-rule="" dir="in">

download app from s3 and unzip
package_file = "#{node['app']['deploy']['dist']}\\#{node['app']['package_name']}"

Chef::Log.info("****** fatch app zip from s3 dist******")  #ec2 with IAM role have s3 access policy , don't set access key
aws_s3_file package_file do
    if aws_access_key_id && aws_secret_access_key
      aws_access_key_id aws_access_key_id
      aws_secret_access_key aws_secret_access_key
    end
    
    bucket node['app']['deploy']['s3_bucket']
    remote_path node['app']['deploy']['s3_key']
    backup false
    action :create
end

deploy_dir = node['app']['deploy']['directory'] # tomcat folder

Chef::Log.info("****** unzip app to local folder path ******")
windows_zipfile deploy_dir do
  source package_file
  overwrite true
  action :unzip
end

AWS CloudFormation Tool

VisualOps

https://www.visualops.io/
How to create a Wordpress app https://www.youtube.com/watch?v=NlWfvl2Cg-M

JSON Editor Online

http://www.jsoneditoronline.org/

Codeship CI/CD for AWS Opswork script

Test

setup

jdk_switcher use oraclejdk8
echo $JAVA_HOME
echo $JRE_HOME
java -version
mvn --version

Configure Test

mvn -B package site -Pqa

Deployment

custom script - do update aws cli and get rev for build package

pip install botocore --upgrade
pip install awscli --upgrade
VERSION=`git rev-parse --short HEAD`
cd target
ZIP_FILE=$(ls -rt *.zip | tail -1)
ZIP_FILE_VERSION=${ZIP_FILE%.*}-${VERSION}.zip
echo "$ZIP_FILE_VERSION" "$ZIP_FILE"
cd ..

S3 - put to AWS S3 - two file, version file and latest

Local Path:./target/${ZIP_FILE}
S3 Bucket:backetabc/dist/${ZIP_FILE_VERSION}

Local Path:./target/${ZIP_FILE}
S3 Bucket:backetabc/dist/${ZIP_FILE}

custom script again - trigger aws opswork

# rolling deployment
git clone https://github.com/awslabs/reinvent2014-scalable-site-management.git
cd reinvent2014-scalable-site-management/opsworks-easy-deploy
pip install -r requirements.txt
python easy_deploy.py --opsworks-region us-east-1 --elb-region us-west-2 deploy --application=root rolling --stack-name={stack name} --layer-name='${layer name}' --comment="Rolling deployment to all web api servers" --timeout 600

2015年11月28日星期六

Use Docker for maven

New container each time

$> docker run -it --rm --name [containername] \
 -v "$PWD":/usr/src/mavenproject -w /usr/src/mavenproject maven:3.3 mvn clean package

Keep container and maven local repository cache (no --rm)

docker run -it  --name [containername]  -v "$PWD":/usr/src/mavenproject -w /usr/src/mavenproject maven:3.3

Mount local repository

docker run -it --rm -v "$PWD":/usr/src/myweb -v "/Users/teman/.m2":/root/.m2 -w /usr/src/myweb  maven:3.3  mvn clean package

Access shell

docker start [containername]
docker exec -it [containername] mvn clean package

Reference

https://hub.docker.com/_/maven/

2015年11月27日星期五

Use Docker for Mysql

https://hub.docker.com/_/mysql/

Run a Mysql container

how to run
$ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --name
Assign a name to the container
-d
Run container in background and print container ID
ex . docker run --name test-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql

expose port

docker run -p 3306:<host port> --name test-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql

expose port and set user/password and database

docker run -p 3306:3306 --name test-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -e MYSQL_DATABASE=mydb -e MYSQL_USER=myuser -e MYSQL_PASSWORD=mypw -d mysql

Manage running container

docker ps -a (list all containers)
docker rm $(docker ps -a -q) (remove all containers)
docker stop $(docker ps -a -q) (stop all containers)

Link Mysql for App

$ docker run --name some-app --link some-mysql:mysql -d application-that-uses-mysql

Connect to MySQL from the MySQL command line client

Description: start a mysql container linked to test-mysql and start mysql client to connect to test-mysql, remove it after exit (--rm)

$ docker run -it --link some-mysql:mysql --rm mysql sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'

-t Allocate a pseudo-TTY
-i Keep STDIN open even if not attached

Container shell access and viewing MySQL logs

$ docker exec -it some-mysql bash

write custom mysql dockerfile and build image and publish

Write a Dockerfile

build image : http://docs.docker.com/mac/step_four/
manager your images and publish: http://docs.docker.com/mac/step_six/

Create a Dockerfile

sample: https://github.com/Chehao/docker-practice/blob/master/mytomcatbuild/Dockerfile

Docker Build

$> docker build -t docker-whale .command takes the Dockerfile in the current directory, and builds an image called docker-whale on your local machine.

Tag it

tag:
docker tag fa8293e27eb1 [namespace]/docker-whale:latest

Pulish to Docker Hub:

docker login --username=yourhubusername --email=youremail@company.com
docker push [namespace]/docker-whale

Run it

$> docker run -e AAA=BBB docker-whale

Docker Command line 基本指令

Commands:

attach Attach to a running container

build Build an image from a Dockerfile

commit Create a new image from a container's changes

cp Copy files/folders between a container and the local filesystem

create Create a new container

diff Inspect changes on a container's filesystem

events Get real time events from the server

exec Run a command in a running container

export Export a container's filesystem as a tar archive

history Show the history of an image

images List images

ex.

~$ docker images

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

mysql 5.7 0f1a7d5b17f5 7 days ago 359.8 MB

mysql latest 0f1a7d5b17f5 7 days ago 359.8 MB

hello-world latest 975b84d108f1 6 weeks ago 960 B

docker/whalesay latest ded5e192a685 6 months ago 247 MB

import Import the contents from a tarball to create a filesystem image

info Display system-wide information

inspect Return low-level information on a container or image

kill Kill a running container

load Load an image from a tar archive or STDIN

logout Log out from a Docker registry

logs Fetch the logs of a container

network Manage Docker networks

pause Pause all processes within a container

port List port mappings or a specific mapping for the CONTAINER

ps List containers

pull Pull an image or a repository from a registry

push Push an image or a repository to a registry

rename Rename a container

restart Restart a container

rm Remove one or more containers 移除 container，但必須先停止 stop

rmi Remove one or more images

run Run a command in a new container 執行一個新的container

ex. docker run --name [some-mysql] -e MYSQL_ROOT_PASSWORD=[my-secret-pw] -d [mysql:tag]
description:

some-mysql : container name
my-secret-pw : mysql root password
mysql:tag : image and tag

save Save an image(s) to a tar archive

search Search the Docker Hub for images

start Start one or more stopped containers

stats Display a live stream of container(s) resource usage statistics

stop Stop a running container 停止執行container

tag Tag an image into a repository

top Display the running processes of a container

unpause Unpause all processes within a container

version Show the Docker version information

volume Manage Docker volumes

wait Block until a container stops, then print its exit code

Docker on Mac Quickstart

Get full steps on http://docs.docker.com/mac/started/

Install Docker Toolbox https://www.docker.com/docker-toolbox
Run Docker Quickstart Terminal
docker hub https://hub.docker.com/

cmd

docker run hello-world

docker run docker/whalesay cowsay boo

docker images

To Know

installed Docker

run a software image in a container

located an interesting image on Docker Hub

run the image on your own machine

modified an image to create your own and run it

create a Docker Hub account and repository

pushed your image to Docker Hub for others to share

訂閱：文章 (Atom)

2015年12月9日 星期三

2015年12月3日 星期四