AMI Blue/Green Deployment

Updated 5 days ago by Michael Katz

This guide outlines a typical configuration and execution of an AMI (Amazon Machine Image) Blue/Green deployment, in the following sections:

Intended Audience

  • DevOps

Before You Begin

Overview

A Blue/Green deployment reliably deploys your AMI(s) by maintaining new and old versions of Auto Scale Groups (ASGs) that are built using these AMIs. The ASGs run behind an Application Load Balancer (ALB) using two listeners, Stage and Prod. These listeners forward respectively to two Target Groups (TGs), Stage and Prod, where the new and old ASGs are run. 

In the first stage of deployment, the new ASG—created using the new AMI you are deploying—is attached to the Stage Target Group:

Blue/Green deployments are achieved by swapping routes between the Target Groups—always attaching the new ASG first to the Stage Target Group, and then to the Prod Target Group:

In Amazon Web Services, you configure a base Launch Configuration that Harness will use when it creates new Launch Configurations; a base Auto Scaling Group that uses the base Launch Configuration; and the Stage and Prod Target Groups. In Harness, you identify the Region, base Auto Scaling Group, and Stage and Prod Target Groups that you've configured in AWS.

This guide outlines the required setup in both AWS and Harness.

Prerequisites

An AMI Blue/Green deployment requires you to set up the following resources up within AWS (example setup below):

  • A working AMI that Harness will use to create the instances in the new ASGs that Harness creates.
  • An AWS Launch Configuration, whose security group allows inbound access to your Application Load Balancer's listener ports.
  • An Auto Scaling Group (ASG), which Harness uses as a template for the ASGs that Harness creates.
  • A pair of Target Groups—typically staging (Stage) and production (Prod)—both with the instance target type.
  • An Application Load Balancer (ALB), with listeners for both your Target Groups' ports.

Within Harness, you'll need to set up the following resources (some of which you might have already created for an AMI Basic deployment):

You do not need to register instances for your Target Groups. Harness will perform that step during deployment.

AWS Setup (Example)

For the Workflow demonstrated here, we set up the following AWS resources:

Launch Configuration

This defines a base Launch Configuration, from which Harness will create new Launch Configurations for new Auto Scaling Groups. This Launch Configuration's security group allows inbound HTTP traffic on port 80 (which we'll use for the Prod Target Group's instance listener).

Auto Scaling Group

The Auto Scaling Group that you define in AWS must use the base Launch Configuration created above. When you later select this ASG in your Harness Service Infrastructure, it becomes the base ASG from which Harness will create new ASGs to deploy new AMIs.

Our example specifies three subnets and modest scaling policies: one instance to start, then Keep this group at its initial size.

Note that if you choose to instead configure scaling policies for your base ASG, Harness will apply these scaling policies in your Workflow's final Upgrade AutoScaling Group step. (Harness will also oblige these policies in any rollback steps.)

Harness specifically supports AWS target tracking scaling policies. For details, see AWS' Dynamic Scaling for Amazon EC2 Auto Scaling topic.

Target Groups

We have a pair of identically configured Target Groups. We've arbitrarily named one TG to indicate production, and the other to indicate staging. This naming convention is a convenience to help us select these TGs when we later assign them to a Harness Service Infrastructure.

Both Target Groups are configured as Target type: Instance. They provide HTTP access for the load balancer on port 80, and specify the Default VPC.

Application Load Balancer (ALB)

The Application Load Balancer is configured with the same Default VPC, and the same subnets, used for the base ASG and the Target Groups.

It has two listeners, forwarding to the two Target Groups: one for production traffic, pointing to the Prod TG; and one for staging traffic, pointing to the Stage TG. In our example, the production Target Group's listener uses port 80, matching the access we configured for that TG.

Add a Blue/Green Service Infrastructure

On the Harness side, the AMI Blue/Green deployment can rely on the same Service, Cloud Provider, and Environment demonstrated in the AMI Basic deployment documentation. However, it requires an expanded Service Infrastructure that specifies both Target Groups. To create this new Service Infrastructure:

  1. Within your Harness Application, select an existing Environment, or create a new one.
  2. In the Environment's Service Infrastructure section, click Add Service Infrastructure.
  3. In the resulting Service Infrastructure section, select an existing Service, or create a new one.
  4. Select a Cloud Provider that references your Delegate by Tag, as outlined earlier for Basic deployment.

    Your dialog will now look something like this:
  5. Click Next to open the dialog's Configuration section.
This section's drop-down lists take a few seconds to populate. Later, some fields will take a few seconds to repopulate, based on your initial selections.
  1. Select the Region and base Auto Scaling Group that you configured in AWS for Blue/Green.
  2. In the upper Target Groups (for ALB) field, select the Target Group that you configured in AWS as your production group.
  3. In the lower Temporary Routes > Target Groups field, select the Target Group that you configured in AWS as your staging group. (Harness uses this Target Group for initial deployment of your service. Upon successful deployment, it swaps this group's route with the production Target Group's route.)

    A completed Configuration section will look something like this:
  4. Click SUBMIT to add the new Service Infrastructure to your Harness Environment.

You're now ready to create a Blue/Green deployment Workflow.

Infrastructure Provisioners

Harness Terraform and CloudFormation Infrastructure Provisioners support Blue/Green deployments for AMI.

When you set up the Service Infrastructure for your Blue/Green deployment, you simply select the option Dynamically Provisioned, then select the Terraform or CloudFormation Infrastructure Provisioner you have set up in your Harness Application. In the following example, you can see a CloudFormation Infrastructure Provisioner selected in Dynamically Provisioned:

The Service Mappings in the Infrastructure Provisioner map the CloudFormation template fields containing the stage and prod Target Groups for Blue/Green deployment.

The AMI Service Mappings act the same as the Already Provisioned fields in the Service Infrastructure you configure manually:

For more information, see CloudFormation Provisioner and Terraform Provisioner.

Create a Blue/Green Workflow

By default, Harness AMI Blue/Green Workflows have five steps:

  1. Setup AutoScaling Group: Specify how many instances to launch, their resizing order, and their steady state timeout.
  2. Deploy Service: Specify the number or percentage of instances to deploy within the ASG you've configured in Setup AutoScaling Group.
  3. Verify Staging: Optionally, specify Verification Providers or Collaboration Providers.
  4. Swap Routes: Re-route requests to the newest stable version of your ASG.
  5. Wrap Up: Optionally, specify post-deployment commands, notifications, or integrations.

Harness preconfigures the Setup, Deploy, and Swap Routes steps. Below, we outline those steps' defaults and options, with examples of the deployment logs' contents at each step.

The Verify Staging and Wrap Up steps are placeholders, to which you can add integrations and commands. For details on adding Verify Staging integrations, see Continuous Verification.

To Create the Blue/Green Workflow:

  1. In your Application, click Workflows > Add Workflow. The Workflow dialog appears.
  2. Enter a Name, and (optionally) enter a Description of this Workflow's purpose.
  3. In Workflow Type, select Blue/Green Deployment.
  4. Select the Environment and Service that you created for your AMI deployment.
  5. Select the Service Infrastructure you configured earlier for AMI Blue/Green deployment.

    The dialog will now look something like this:
  6. Click SUBMIT. The new Blue/Green Workflow for AMI is preconfigured.

Next, we will examine options for configuring the Blue/Green deployment's Setup, Deploy, and Swap Routes steps.

Step 1: Setup AutoScaling Group

In Step 1, select AWS AutoScaling Group Setup to open a dialog where you can fine-tune the new Auto Scaling Group (ASG) that Harness creates for the AMI Service you are deploying:

For most settings here, see the corresponding AMI Basic Workflow instructions. However:

Harness recommends setting the Auto Scaling Steady State Timeout (mins) field to at least 20 minutes, as shown above. This is a safe interval to prevent failed deployments while the Swap Routes step's Blue/Green switchover completes.

Setup AutoScaling Group in Deployment

Let's look at an example where the AWS AutoScaling Group Setup—configured as shown above—is deployed. Here is the step in the Harness Deployments page:

Here's partial output, showing a successful setup:

Starting AWS AMI Setup
Starting AWS AMI Setup
Getting base auto scaling group
Getting base launch configuration
Getting all Harness managed autoscaling groups
Getting last deployed autoscaling group with non zero capacity
Creating new launch configuration [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]
Creating new AutoScalingGroup [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]
Sending request to delete old auto scaling groups to executor
Completed AWS AMI Setup with new autoScalingGroupName [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]

Step 2: Deploy Service

In Step 2, select Upgrade AutoScaling Group to open a dialog where you can define how many instances to deploy in the Auto Scaling Group, as either a count or a percentage.

For general information on customizing this dialog's settings, and on how they correspond to AWS parameters, see the corresponding AMI Basic Workflow section. This deployment example uses percentage scaling, with a desired target of 100%.

If your base Auto Scaling Group is configured in AWS with scaling policies, Harness will apply those policies in your Workflow's final Upgrade AutoScaling Group step.

Deploy Service Step in Deployment

At this point, Harness deploys the new ASG—containing the instances created using your new AMI—to the Stage Target Group:

Using the Upgrade AutoScaling Group configuration shown above, here is the Deploy Service step in the Harness Deployments page:

Here is partial output, showing the new Auto Scaling Group successfully resized and at steady state:

Starting AWS AMI Deploy
Getting existing instance Ids
Resizing Asgs
Resizing AutoScaling Group: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2] to [1]
Set AutoScaling Group: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2] desired capacity to [1]
Successfully set desired capacity
AutoScalingGroup [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2] activity [Launching a new EC2 instance: i-05aace750dbed65b3] progress [30 percent] , statuscode [PreInService] details [{"Subnet ID":"subnet-e962248d","Availability Zone":"us-east-1a"}]
Waiting for instances to be in running state. pending=1
AutoScalingGroup [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2] activity [Launching a new EC2 instance: i-05aace750dbed65b3] progress [30 percent] , statuscode [PreInService] details [{"Subnet ID":"subnet-e962248d","Availability Zone":"us-east-1a"}]
AutoScaling group reached steady state
Setting min capacity of Asg[AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2] to [1]

Next, the staging Target Group is attached to the new ASG, and its target instances are registered:

Waiting for Target Group: [arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-stage/8e281748dd2c6344] to have all instances of Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]
[0] out of [1] targets registered and in healthy state
[...]
AutoScaling Group resize operation completed with status:[SUCCESS]
[1] out of [1] targets registered and in healthy state
All targets registered for Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]

Approval Sub-Step

This example shows an (optional) Approval added to Step 2. It requests manual approval, following successful registration of the staging group, and prior to the Blue/Green (staging/production) switchover in the next step.

Step 4: Swap Routes

This is the heart of a Blue/Green deployment. Here, Harness directs the Application Load Balancer to:

  • Detach your staging Target Group from the new ASG.
  • Attach your production Target Group to the new ASG, to handle incoming requests.
  • Detach your production Target group from the old ASG.

When this step is complete, the new ASG—containing the instances created using your new AMI—are deployed to the production TG:

In Step 4, open the Switch Auto Scaling Group Route dialog if you want to toggle the Downsize Old Auto Scaling Group setting. When enabled, this check box directs AWS to free up resources from the old ASG once the new ASG registers its targets and reaches steady state.

Switch Auto Scaling Group Route Step in Deployment

Using the configuration shown above, here is the Switch Auto Scaling Group Route step in the Harness Deployments page:

Here's partial output, showing successful swapping of the two Target Groups' routes. First, the staging Target Group is detached from new ASG 2:

Starting to switch routes in AMI Deploy
Starting Ami B/G swap
Sending request to detach target groups:[arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-stage/8e281748dd2c6344] from Asg:[AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]
Waiting for Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2] to de register with target group: [arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-stage/8e281748dd2c6344]
[1] out of [1] targets still registered
[...]
All targets de-registered for Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]

Next, the production Target Group is attached to ASG 2. Then, its targets are verified healthy and registered. This new ASG now handles incoming requests:

Sending request to attach target groups:[arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-prod/28cfdfd155415a62] to Asg:[AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]
Waiting for Target Group: [arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-prod/28cfdfd155415a62] to have all instances of Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]
[1] out of [1] targets registered and in healthy state
All targets registered for Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__2]

Next, the production group is detached from ASG 1:

Sending request to detach target groups:[arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-prod/28cfdfd155415a62] from Asg:[AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1]
Waiting for Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1] to deregister with target group: [arn:aws:elasticloadbalancing:us-east-1:XXXXXXXXXXXX:targetgroup/bg-doc-prod/28cfdfd155415a62]
All targets de-registered for Asg: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1]

Finally, the old ASG is downsized to 0 instances. Had the Workflow's Step 4 (Swap Routes) not specified Downsize Old Auto Scaling Group, these resources would not be explicitly freed up:

Downscaling autoScaling Group [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1]
Set AutoScaling Group: [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1] desired capacity to [0]
Successfully set desired capacity
AutoScalingGroup [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1] activity [Terminating EC2 instance: i-01dc003477e21fcb6] progress [100 percent] , statuscode [Successful] details [{"Subnet ID":"subnet-e962248d","Availability Zone":"us-east-1a"}]
AutoScalingGroup [AMI__Blue__Green__Application_AMI__Blue__Green__Service_AWS__Blue__Green__Doc__1] activity [Launching a new EC2 instance: i-01dc003477e21fcb6] progress [100 percent] , statuscode [Successful] details [{"Subnet ID":"subnet-e962248d","Availability Zone":"us-east-1a"}]
AutoScaling group reached steady state
Completed switch routes

Blue/Green Workflow Deployment

As with the AMI Basic deployment, once your setup is complete, you can click the Workflow's Deploy button to start the Blue/Green deployment.

In the resulting Start New Deployment dialog, select the AMI to deploy, and click SUBMIT.

The Workflow deploys. The Deployments page displays details about the deployed instances.

To verify the completed deployment, log into your AWS Console and locate the newly deployed instance(s).

Troubleshooting

The following errors might occur when you run an AMI Blue/Green deployment in Harness.

Valid Blue/Green Deployment Failed and Rolled Back in Harness

This can occur when Harness' steady state timeout setting is too restrictive, compared to the time AWS requires to swap your Target Groups' routes.

To resolve the rollbacks: In your Blue/Green Workflow's Step 1 (Setup AutoScaling Group), try raising the Auto Scaling Steady State Timeout (mins) setting to at least match the switchover interval you observe in the AWS Console.

Next Steps


How did we do?