MongoDB Clustor

MongoDB Clustor

• Router (mongos) X2

sudo -u mongo /data/mongodb-3.0.0/bin/mongos --port 27017 --configdb CNF01:27017,CNF02:27017,CNF03.SJC1:27017 \ 
--logpath /data/logs/ROUTER.log --logappend --fork

• Config DB Server X 3

/data/mongodb-3.0.0/bin/mongod --port 27017 --configsvr --dbpath /data/dbs/CNF \ 
--logpath /data/logs/CNF.log --logappend --fork

• SRD X3 , ReplicaSet X3

ulimit -n 65535
sudo -u mongo  numactl --interleave=all /data/mongodb-3.0.0/bin/mongod --port 27001 --replSet ReplicaSet_01 --dbpath /data/dbs/SRD1  \
--logpath /data/logs/SRD1.log --logappend --fork   

• Start with config file

$> mongod –f mongodb.conf

Setup

• ReplicaSet

rs.initiate()
rs.conf()
rs.add("SRD2:27017")
rs.addArb("ARB1:27017")

Shards

• Login to Router (mongos)

mongo> sh.addShard("ReplicaSet_01/SRD1:27017") # auto add rest replica

• Enable shard

mongo> sh.enableSharding("Db")
sh.shardCollection("Db.collection", key: {"key":1})

2015-12-1 Read Note

2015-12-1 Read Note

• 10 Awesome Docker Tutorials to Kick-Start your DevOps Projects
• Next Generation Eclipse IDE : Che naming by me ?XD I would try it with docker container tomorrow 

AWS Opswork for windows , customize layer / recipes

AWS Opswork stack , 

  if you want to use windows , custom layer  only support windows 2012

required cookbook

• aws Cookbook https://github.com/chef-cookbooks/aws
• Windows Cookbook https://github.com/chef-cookbooks/windows
•  java https://github.com/agileorbit-cookbooks/java

write your recipe

install java , setting custom json and set receipe to custom layer, "java"

{
  "java" : {
      "install_flavor": "windows", 
      "package_name":"Java SE Development Kit 8 Update 20 (64-bit)",
      "url" : "s3://backet/XXXX"
  }
}

install awscli for windows (I don't like powershell) default.rb

if kernel['machine'] =~ /x86_64/
  default['awscli']['url']          = "https://s3.amazonaws.com/aws-cli/AWSCLI64.msi"
else
  default['awscli']['url']          = "https://s3.amazonaws.com/aws-cli/AWSCLI32.msi"  
end

default['awscli']['checksum'] = nil
default['awscli']['package_name'] = "AWS Command Line Interface"

your_recipe.rb

pkg_checksum = node['awscli']['checksum']

windows_package node['awscli']['package_name'] do
  source node['awscli']['url']  
  checksum pkg_checksum if pkg_checksum 
  action :install
end

install .net3.5 feature for windows, default windows 2012 only .net4

Chef::Log.info("****** install .Net3.5 feature  ******")
windows_feature "NetFx3" do
  action :install
  all true
end  

install tomcat

pkg_checksum = node['tomcat']['checksum']

windows_package node['tomcat']['package_name'] do 
  source node['tomcat']['url']
  checksum pkg_checksum if pkg_checksum 
  action :install
end

windows_service node['piston']['tomcat']['service_name'] do
  action :configure_startup
  startup_type :automatic
end

Chef::Log.info("******Add firewall rule******")
#TO-DO: need to modify port as parameter
batch "Add firewall rule for Tomcat Server" do
  code <<-eoh data-blogger-escaped-action="allow" data-blogger-escaped-add="" data-blogger-escaped-advfirewall="" data-blogger-escaped-end="" data-blogger-escaped-eoh="" data-blogger-escaped-firewall="" data-blogger-escaped-localport="8080" data-blogger-escaped-name="Open Port 8080" data-blogger-escaped-netsh="" data-blogger-escaped-pre="" data-blogger-escaped-protocol="TCP" data-blogger-escaped-rule="" dir="in">

download app from s3 and unzip
package_file = "#{node['app']['deploy']['dist']}\\#{node['app']['package_name']}"

Chef::Log.info("****** fatch app zip from s3 dist******")  #ec2 with IAM role have s3 access policy , don't set access key
aws_s3_file package_file do
    if aws_access_key_id && aws_secret_access_key
      aws_access_key_id aws_access_key_id
      aws_secret_access_key aws_secret_access_key
    end
    
    bucket node['app']['deploy']['s3_bucket']
    remote_path node['app']['deploy']['s3_key']
    backup false
    action :create
end

deploy_dir = node['app']['deploy']['directory'] # tomcat folder

Chef::Log.info("****** unzip app to local folder path ******")
windows_zipfile deploy_dir do
  source package_file
  overwrite true
  action :unzip
end

AWS CloudFormation Tool

VisualOps

• https://www.visualops.io/
• How to create a Wordpress app https://www.youtube.com/watch?v=NlWfvl2Cg-M 

JSON Editor Online

• http://www.jsoneditoronline.org/

Codeship CI/CD for AWS Opswork script

Test

setup

jdk_switcher use oraclejdk8
echo $JAVA_HOME
echo $JRE_HOME
java -version
mvn --version

Configure Test

mvn -B package site -Pqa

Deployment

custom script - do update aws cli and get rev for build package

pip install botocore --upgrade
pip install awscli --upgrade
VERSION=`git rev-parse --short HEAD`
cd target
ZIP_FILE=$(ls -rt *.zip | tail -1)
ZIP_FILE_VERSION=${ZIP_FILE%.*}-${VERSION}.zip
echo "$ZIP_FILE_VERSION" "$ZIP_FILE"
cd ..

S3 - put to AWS S3 - two file, version file and latest

Local Path:./target/${ZIP_FILE}
S3 Bucket:backetabc/dist/${ZIP_FILE_VERSION}

Local Path:./target/${ZIP_FILE}
S3 Bucket:backetabc/dist/${ZIP_FILE}

custom script again - trigger aws opswork

# rolling deployment
git clone https://github.com/awslabs/reinvent2014-scalable-site-management.git
cd reinvent2014-scalable-site-management/opsworks-easy-deploy
pip install -r requirements.txt
python easy_deploy.py --opsworks-region us-east-1 --elb-region us-west-2 deploy --application=root rolling --stack-name={stack name} --layer-name='${layer name}' --comment="Rolling deployment to all web api servers" --timeout 600

Use Docker for maven

New container each time

$> docker run -it --rm --name [containername] \
 -v "$PWD":/usr/src/mavenproject -w /usr/src/mavenproject maven:3.3 mvn clean package

Keep container and maven local repository cache (no --rm)

docker run -it  --name [containername]  -v "$PWD":/usr/src/mavenproject -w /usr/src/mavenproject maven:3.3 

Mount local repository 

docker run -it --rm -v "$PWD":/usr/src/myweb -v "/Users/teman/.m2":/root/.m2 -w /usr/src/myweb  maven:3.3  mvn clean package

Access shell

• docker start [containername]
• docker exec -it [containername] mvn clean package

Reference

• https://hub.docker.com/_/maven/

Use Docker for Mysql

• https://hub.docker.com/_/mysql/

Run a Mysql container

• how to run
  $ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --name
  
  Assign a name to the container
  -d  
  Run container in background and print container ID
  ex . docker run --name test-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql

  

expose port

• docker run -p 3306:<host port> --name test-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql

expose port and set user/password and database

• docker run -p 3306:3306 --name test-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -e MYSQL_DATABASE=mydb -e MYSQL_USER=myuser -e MYSQL_PASSWORD=mypw -d mysql

Manage running container 

• docker ps -a (list all containers)
• docker rm $(docker ps -a -q)   (remove all containers)
• docker stop $(docker ps -a -q) (stop all containers)

Link Mysql for App

• $ docker run --name some-app --link some-mysql:mysql -d application-that-uses-mysql

Connect to MySQL from the MySQL command line client

    Description: start a mysql container linked to test-mysql and start mysql client to connect to test-mysql,  remove it after exit (--rm)

• $ docker run -it --link some-mysql:mysql --rm mysql sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
• -t Allocate a pseudo-TTY
• -i Keep STDIN open even if not attached

Container shell access and viewing MySQL logs

• $ docker exec -it some-mysql bash

write custom mysql dockerfile and build image and publish