6 - Kubernetes Blue/Green Workflows

Updated 1 month ago by Michael Cretzman

This topic describes how to create a Harness Service and Workflow for a Blue/Green Kubernetes deployment.

There are no Blue/Green-specific settings for the Harness Environment (see Kubernetes Environments for steps on setting up the Harness Environment for any Kubernetes deployment).

Blue/Green Deployment Summary

Blue/Green deployment is a method that reduces downtime and risk by running two identical environments called Blue and Green. At any time, only one of the environments is live, serving all production traffic. For example, Blue is live (primary) and Green is idle (stage).

As you prepare a new version of your software, deployment and the final stage of testing takes place in the stage environment, for example Green. Once you have deployed and fully tested the software in Green, you switch the network routing so all incoming requests now go to Green instead of Blue. Green is now live, and Blue is idle. In Kubernetes, a primary and a stage Kubernetes service are swapped.

For information on Kubernetes services, see Services from Kubernetes.

Harness Canary and Blue/Green Workflow types only support Kubernetes Deployment workloads. Rolling Workflow types support all other workloads, except Jobs. The Apply Step can deploy any workloads or objects.

Blue/Green Rollback

A great benefit of a Blue/Green deployment is rapid rollback: rolling back to the old version of a service/artifact is simple and reliable because network traffic is simply routed back to the original instances. You do not need to redeploy previous versions of the service/artifact and the instances that comprised their environment.

Kubernetes Service for Blue/Green

When you create a Harness Service for a Blue/Green deployment, you need to include a specification for both of the Kubernetes services using in Blue/Green.

Harness refers to the two services as primary and stage Kubernetes services, distinguished using the following mandatory annotations:

  • Primary - annotations: harness.io/primary-service: true
  • Stage - annotations: harness.io/stage-service: true

No other labels or values are need to distinguish the two Kubernetes services.

Here is an example Harness Service Manifests section for Blue/Green.

You can see the primary Kubernetes service, service-green.yaml, and the stage service, service-blue.yaml. The spec.yaml file contains the Deployment object manifest, and the values.yaml is used for Go templating.

Let's quickly look at the Service Manifests files for a Blue/Green deployment. If you have not created a Kubernetes type Harness Service, see Kubernetes Services.

Here is the values.yaml:

name: harness-example
replicas: 3
image: ${artifact.metadata.image}

Here is the service-green.yaml (primary) manifest file:

apiVersion: v1
kind: Service
metadata:
name: {{.Values.name}}-svc-primary

# mandatory annotation indicating primary service

annotations:
harness.io/primary-service: true
labels:
app: bg-demo-app
spec:
type: ClusterIP
ports:
- port: 80
protocol: TCP
selector:
app: bg-demo-app

Here is the service-blue.yaml (stage) manifest file:

apiVersion: v1
kind: Service
metadata:
name: {{.Values.name}}-svc-stage

# mandatory annotation indicating stage service

annotations:
harness.io/stage-service: true
labels:
app: bg-demo-app
spec:
type: ClusterIP
ports:
- port: 80
protocol: TCP
selector:
app: bg-demo-app

Here is the deployment.yaml that contains the Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
name: {{.Values.name}}
spec:
selector:
matchLabels:
app: bg-demo-app
replicas: {{.Values.replicas}}
template:
metadata:
labels:
app: bg-demo-app
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80

That is all that is needed to set up a simple Harness Service for Kubernetes Blue/Green deployment.

Kubernetes Workflows for Blue/Green

When you create a Harness Kubernetes Workflow for Blue/Green deployment, Harness automatically generates the steps for setting up the Kubernetes services you defined in your Harness Service, and for swapping the Kubernetes services between the applications.

To create a Kubernetes Blue/Green Workflow, do the following:

  1. In your Application containing a completed Service and Environment for the Blue/Green deployment, click Workflows.
  2. Click Add Workflow. The Workflow dialog appears.
  3. In Name, enter a name for your Workflow, such as NGINX-K8s-BG.
  4. In Workflow Type, select Blue/Green Deployment.
  5. In Environment, select the Environment you created for your Kubernetes deployment.
  6. In Service, select the Service containing the manifest files you want to use for your deployment.
  7. In Service Infrastructure/Infrastructure Definition, select the Service Infrastructure/Infrastructure Definition where you want to deploy.
    When you are finished the Workflow dialog will look like this (this example uses the feature-flagged Infrastructure Definition example):
  8. Click SUBMIT. The new Workflow appears.

Let's look at each step in the Workflow and its deployment step log.

Stage Deployment

The Stage Deployment step is added automatically when you create the Workflow:

Skip Dry Run: By default, Harness uses the --dry-run flag on the kubectl apply command during the Initialize step of this command, which prints the object that would be sent to the cluster without really sending it. If the Skip Dry Run option is selected, Harness will not use the --dry-run flag.

The Stage Deployment step simply deploys the two Kubernetes services you have set up in the Harness Service Manifests section.

When you look at the Stage Deployment step in Harness Deployments, you will see the following log sections.

Initialize

The Initialize stage initializes the two Kubernetes services you have set up in the Harness Service Manifests section (displayed earlier), primary and stage, validating their YAML.

Initializing..

Manifests [Post template rendering] :

---
apiVersion: v1
kind: Service
metadata:
name: harness-example-svc-primary
annotations:
harness.io/primary-service: true
labels:
app: bg-demo-app
spec:
type: ClusterIP
ports:
- port: 80
protocol: TCP
selector:
app: bg-demo-app
---
apiVersion: v1
kind: Service
metadata:
name: harness-example-svc-stage
annotations:
harness.io/stage-service: true
labels:
app: bg-demo-app
spec:
type: ClusterIP
ports:
- port: 80
protocol: TCP
selector:
app: bg-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: harness-example
spec:
selector:
matchLabels:
app: bg-demo-app
replicas: 3
template:
metadata:
labels:
app: bg-demo-app
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80


Validating manifests with Dry Run

kubectl --kubeconfig=config apply --filename=manifests-dry-run.yaml --dry-run
service/harness-example-svc-primary configured (dry run)
service/harness-example-svc-stage configured (dry run)
deployment.apps/harness-example created (dry run)

Done.
Prepare

Typically, in the Prepare section, you can see that each release of the resources is versioned. This is used in case Harness needs to rollback to a previous version. In the case of Blue/Green, the resources are not versioned because a Blue/Green deployment uses rapid rollback: network traffic is simply routed back to the original instances. You do not need to redeploy previous versions of the service/artifact and the instances that comprised their environment.

Manifests processed. Found following resources: 

Kind Name Versioned
Service harness-example-svc-primary false
Service harness-example-svc-stage false
Deployment harness-example false

Primary Service is at color: blue
Stage Service is at color: green

Cleaning up non primary releases

Current release number is: 2

Versioning resources.

Workload to deploy is: Deployment/harness-example-green

Done.
Apply

The Apply section applies a combination of all of the manifests in the Service Manifests section as one file using kubectl apply.

kubectl --kubeconfig=config apply --filename=manifests.yaml --record

service/harness-example-svc-primary configured
service/harness-example-svc-stage configured
deployment.apps/harness-example-blue configured

Done.
Wait for Steady State

The Wait for Steady State section displays the blue service rollout event.

kubectl --kubeconfig=config get events --output=custom-columns=KIND:involvedObject.kind,NAME:.involvedObject.name,MESSAGE:.message,REASON:.reason --watch-only

kubectl --kubeconfig=config rollout status Deployment/harness-example-blue --watch=true


Status : deployment "harness-example-blue" successfully rolled out

Done.

Next, the Swap Primary with Stage Workflow step will swap the blue and green services to route primary network traffic to the new version of the application, and stage network traffic to the old version.

Swap Primary with Stage

In the Blue/Green Workflow, click the Swap Primary with Stage step.

You can see that the primary Kubernetes service is represented by the variable ${k8s.primaryServiceName}, and the stage service by the variable ${k8s.stageServiceName}. You can see how the swap works in the Swap Primary with Stage step in Harness Deployments.

Here is the log for the step, where the mandatory Selectors you used in the Harness Service Manifests files are used.

Begin execution of command Kubernetes Swap Service Selectors

Selectors for Service One : [name:harness-example-svc-primary]
app: bg-demo-app
harness.io/color: green


Selectors for Service Two : [name:harness-example-svc-stage]
app: bg-demo-app
harness.io/color: blue


Swapping Service Selectors..

Updated Selectors for Service One : [name:harness-example-svc-primary]
app: bg-demo-app
harness.io/color: blue


Updated Selectors for Service Two : [name:harness-example-svc-stage]
app: bg-demo-app
harness.io/color: green

Done
The Swap Primary with Stage command is simply the Swap Service Selectors command renamed to Swap Primary with Stage for this Workflow type. You can use Swap Service Selectors to swap the pods referred to by any two Kubernetes services. You simply put the expressions for any two services (${k8s.primaryServiceName}${k8s.stageServiceName}) and they will be swapped. For example, you can have a Blue/Green deployment Workflow to swap services and then a separate Workflow that uses the Swap Service Selectors command to manually swap back when needed.

Blue/Green Workflow Deployment

Now that the setup is complete, you can click Deploy in the Workflow to deploy the artifact to your cluster.

Next, select the artifact build version and click SUBMIT.

The Workflow is deployed.

The swap is complete and the Blue/Green deployment was a success.

In the Deployments page, expand the Workflow steps and click the Swap Primary with Stage step.

In the Details section, click the vertical ellipsis and click View Execution Context.

You can see that the names and of primary and stage services deployed.

Now that you have successfully deployed your artifact to your Kubernetes cluster pods using your Harness Application, look at the completed workload in the deployment environment of your Kubernetes cluster.

For example, here is the Blue/Green workload in Google Cloud Kubernetes Engine, displaying the blue and green services and Deployment workload:

If you click a workload, you will see the pods and service created:

Next Steps


How did we do?