Blue Green Deployment using AWS Cloudformation and Lambda

One of our client requirement to implement Blue/Green deployment for their application which is using AWS autoscaling.
Here's the solution we have implemented for them.
Blue/Green Deployment reduces downtime and risk during the cut-off. This consist of bringing up an identical green environment as blue (which is currently the "live" environment). Once the testing is done in green environment and ready to go live, all the user's traffic can simply be switched to the green environment.
In this blog post, we have shown the automation of blue/green deployment using AWS Cloudformation and Lambda services. Here we go -->
https://github.com/powerupcloud/bluegreenswitcher Refer to this link for the Lambda code and cloudformation template which we have used for this post.

Deployment Workflow:

  • Create Green environment: Creating a green environment exactly same as blue using a cloudformation template.
  • Test the Green environment: A test Elastic Load Balancer will be attached to green environment using the same template. Testing can be done after the green environment has been setup.
  • Switch the ELB: On Successful testing, switch the Production Elastic Load Balancer from Blue to Green by updating the existing stack.

Prerequisites:

  • Blue Environment:

Assuming we already have a blue environment setup such as an autoscaling group with a production elastic load balancer attached to it.

  • Test Elastic Load Balancer:

We have also setup a test elastic load balancer which can be temporarily attached with green environment to test the application.

  • Lambda Function (To update instances count in green environment same as blue environment):

Create a lambda function to update the green environment with same number of instances as in blue environment. A snippet of this function code is shown below in which we are calling updateAutoScalingGroup API to update the number of instances count of green environment same as blue environment.

 asg.describeAutoScalingGroups(function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred                                                                                                                              
  else {
    for (var i = 0; i < data.AutoScalingGroups.length; i++) {
      for (var j = 0; j < data.AutoScalingGroups[i].LoadBalancerNames.length; j++) {
        if (data.AutoScalingGroups[i].LoadBalancerNames[j] == prodelbname) {
        var asgblue=data.AutoScalingGroups[i].AutoScalingGroupName
        console.log("Blue AutoScaling Group Name: ", asgblue);
        greenMinSize = data.AutoScalingGroups[i].MinSize
        greenDesiredCapacity = data.AutoScalingGroups[i].DesiredCapacity
         greenMaxSize = data.AutoScalingGroups[i].MaxSize
         var greenasgparams = {
             AutoScalingGroupName: asggreen,
             MinSize: greenMinSize,
             DesiredCapacity: greenDesiredCapacity,
             MaxSize: greenMaxSize
         }
      }
   }
 }
 console.log("Updating Green Asg to have equal no. of instances as Blue Asg");
 asg.updateAutoScalingGroup(greenasgparams, function(err, data) {
     if (err) console.log(err, err.stack); // an error occurred
     else console.log(data);
 });
  • Lambda Function (To Switch Production Elastic Load Balancer from Blue Environment to Green):

Create an another lambda function to switch ELB after the application testing in Green environment is done. A snippet of this function code is shown below in which we are calling different APIs to detach the production Elastic loadbalancer from blue environment and attaching it to the green environment. Also, detaching the test elastic loadbalancer from the green environment before attaching the production ELB. And, atlast after switching ELB, terminating the blue environment by calling “updateAutoScalingGroup” API to update the blue Autoscaling size to zero.

asg.describeAutoScalingGroups(function(err, data) {  
 if (err) console.log(err, err.stack); // an error occurred
 else {
  for (var i = 0; i < data.AutoScalingGroups.length; i++) {
    for (var j = 0; j <                data.AutoScalingGroups[i].LoadBalancerNames.length; j++) {
       if (data.AutoScalingGroups[i].LoadBalancerNames[j] == prodelbname) {
          var asgblue=data.AutoScalingGroups[i].AutoScalingGroupName
          console.log("Asg Blue: ", asgblue);
          var bluedetachparams = {
             AutoScalingGroupName: asgblue,
             LoadBalancerNames: [prodelbname]
          }
          var asgparams = {
             AutoScalingGroupName: asgblue,
             MinSize: 0,
             DesiredCapacity: 0,
             MaxSize: 0
          }
       }
    }
  }
  var greendetachparams = {
      AutoScalingGroupName: asggreen,
      LoadBalancerNames: [testelbname]
  }
  console.log("Detaching Test ELB (", testelbname, ") from Green  Autoscaling group (", asggreen, ")");
  asg.detachLoadBalancers(greendetachparams, function(err, data) {
     if (err) console.log(err, err.stack); // an error occurred
     else console.log(data); // successful response
  });
  var greenattachparams = {
     AutoScalingGroupName: asggreen,
     LoadBalancerNames: [prodelbname]
  };
  console.log("Attaching ELB (", prodelbname, ") to Green Autoscaling group (", asggreen, ")");
  asg.attachLoadBalancers(greenattachparams, function(err, data) {
     if (err) console.log(err, err.stack); // an error occurred
     else console.log(data); // successful response
  });
  console.log("Detaching ELB (", prodelbname, ") from Blue Autoscaling group (" + asgblue + ")");
  asg.detachLoadBalancers(bluedetachparams, function(err, data) {
     if (err) console.log(err, err.stack); // an error occurred
     else console.log(data); // successful response
  });
  console.log("Terminating Blue Environment");
  asg.updateAutoScalingGroup(asgparams, function(err, data) {
     if (err) console.log(err, err.stack); // an error occurred
     else {
  • Two SNS Topics subscribed to both the lambda functions respectively.

Since we are using India region for this blue green deployment and AWS Lambda is not available in India region so we are triggering these lambda functions by subscribing them to the SNS topics. So we have two lambda functions in US-EAST-1 which are subscribed to two SNS topics respectively in India region. Also, the sns topics should have proper policy attached with it to receive messages from cloudformation. The policy to be attached is shown below:

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudformation.amazonaws.com"
      },
      "Action": [
        "SNS:GetTopicAttributes",
        "SNS:SetTopicAttributes",
        "SNS:AddPermission",
        "SNS:RemovePermission",
        "SNS:DeleteTopic",
        "SNS:Subscribe",
        "SNS:ListSubscriptionsByTopic",
        "SNS:Publish",
        "SNS:Receive"
      ],
      "Resource": "arn:aws:sns:ap-south-1:<account-number>:<topic-name>"
    }
  ]
}

  • One SNS Topic subscribed to an email which will get the alerts of Autoscaling group.

Launching Green Environment:

Go to Cloudformation and start launching the green environment by creating a stack. Upload the cloudformation template.

We are not going to provide UpdateElbArn for now while creating this stack. So, this stack creation will update the green environment same as blue environment using custom resource which will trigger the lambda function which we setup to achieve this scenario.
Click Next. Review all settings and Create Stack.
So if we check green autoscaling group, after creating it will be having Min Max instances set to 1 initially.
The size of autoscaling group will be updated after custom resource creation which will set the size values to same as blue. Also, the test elastic loadbalancer is attached to green environment.
The final status of the stack will be CREATE_COMPLETE after success.
So, We have now our green environment ready with a test loadbalancer attached to it. The application can be tested in this environment. Now if we want to make this green environment as production, we just need to update the stack and provide the UpdateElbArn which will not only make this green environment production but it will also terminate the existing blue environment.

Switching Prod ELB to Green Environment and Terminating the Blue Environment:

Click on Update Stack.
Use current template and proceed to the next screen. Provide UpdateElbArn in the parameters while keeping other values as it is.
Click next. Preview your changes one more time. Ensure you are ready to switch your production environment.
Update.
It will start creating another custom resource which will switch the Prod Elb to green.
After Update complete, we can see the production elastic load balancer get attached to the green environment and the blue autoscaling group will be terminated.
Now, for the next deployment, assume this environment as blue and start creating stack by uploading the same template and it will create next green deployment environment with a test elb which can be used for testing and after ensuring, can be switched to production environment.

Achieving the same using API Calls:

Below is a bash script which will do the following jobs:

  • Creates Image of the instance to be launched in the green environment
  • The process will wait till the image becomes available
  • Creates cloudformation stack with the above created Image Id passed in parameters.

Here’s is the script:

#!/bin/bash
NOW=`date +'%F-%H%M'`  
aws ec2 create-image --instance-id i-xxxxxxxxxxxx --name CMOLAPI-Base-$NOW --no-reboot > imageid  
IMAGE=`cat imageid | jq '.ImageId' | sed 's/\"//g'`  
aws ec2 wait image-available --image-ids $IMAGE  
if [ $? -eq 0 ]  
then  
echo "Creating CloudFormation Stack"  
aws cloudformation create-stack --stack-name BlueGreen --template-body https://s3.amazonaws.com/cfn-deployments/bluegreen-mumbai.json --parameters ParameterKey=InstanceType,ParameterValue=t2.medium ParameterKey=Subnets,ParameterValue='subnet-cca1fd8e\,subnet-06c5b52f' ParameterKey=SecurityGroup,ParameterValue=sg-f8c90550 ParameterKey=KeyName,ParameterValue=bgkey ParameterKey=Vpc,ParameterValue=vpc-71de3c32 ParameterKey=IAMInstanceProfileName,ParameterValue=baston ParameterKey=UpdateInstanceCountArn,ParameterValue=arn:aws:sns:ap-south-1:111111000000:UpdateInstanceCount ParameterKey=AsgAlarmArn,ParameterValue=arn:aws:sns:ap-south-1:111111000000:test-topicemail-11XSV3I18N2HW ParameterKey=AmazonLinuxAmi,ParameterValue=$IMAGE ParameterKey=ProdElasticLoadBalancerName,ParameterValue=prod-elb ParameterKey=TestElasticLoadBalancerName,ParameterValue=test-elb  
rm imageid  
else  
echo "goodbye"  
fi  

Similarly, to update the stack, refer below command:

aws cloudformation update-stack --stack-name BlueGreen --use-previous-template --parameters ParameterKey=UpdateElbArn,ParameterValue=arn:aws:sns:ap-south-1:111111000000:UpdateElb ParameterKey=InstanceType,UsePreviousValue=true ParameterKey=Subnets,UsePreviousValue=true ParameterKey=SecurityGroup,UsePreviousValue=true ParameterKey=KeyName,UsePreviousValue=true ParameterKey=Vpc,UsePreviousValue=true ParameterKey=IAMInstanceProfileName,UsePreviousValue=true ParameterKey=UpdateInstanceCountArn,UsePreviousValue=true ParameterKey=AsgAlarmArn,UsePreviousValue=true ParameterKey=AmazonLinuxAmi,UsePreviousValue=true ParameterKey=ProdElasticLoadBalancerName,UsePreviousValue=true ParameterKey=TestElasticLoadBalancerName,UsePreviousValue=true  

Integration with Jenkins:

Log into the Jenkins Dashboard:
Create a New Item. Give a name to the Jenkins Job and Select Multi Configuration Project.
Next, Configure the job. Scroll down to add build step.
Select Execute shell and give command to run your bash script available in the server.
Save it. The job will appear on the dashboard. Select Build now to run the script.
After testing the green environment, similarly, build an another job for switching the Prod Elb to green and pass the Update-Stack command in the add build step.

and that's it. Just a single click and you can switch the live environment from blue to the green environment. Hope it will be helpful in automating the deployment. Happy Blue/Green Deployment..!! :)

Priyanka Sharma

Priyanka is Senior Cloud and DevOps Engineer. She can churn out CloudFormation templates at a moment's notice and play with Chef/Ansible. Dancing, music, badminton and word games are her hobbies

comments powered by Disqus