3 - Verify Deployments with Stackdriver

Updated 3 weeks ago by Michael Cretzman

Harness can analyze Stackdriver data and analysis to verify, rollback, and improve deployments. To apply this analysis to your deployments, you set up Stackdriver as a verification step in a Harness Workflow.

This section covers how to set up Stackdriver in a Harness Workflow, and provides a summary of Harness verification results.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

You can use both Stackdriver Logs and Metrics in a Workflow.

Deployment Verification Setup

To verify your deployment with Stackdriver, do the following:

  1. Ensure that you have added Google Cloud Platform as a Cloud Provider provider, as described above.
  2. In your Workflow, under Verify Service, click Add Verification, and then click Stackdriver. The Stackdriver dialog appears.

  1. In GCP Cloud Provider, select the Google Cloud Platform (GCP) Cloud Provider you set up in Harness.
  2. In Region, select the GCP region where the application is hosted. The Stackdriver API uses a service-specific notion of location. Harness uses the name of a region. You can find the region in Stackdriver Metrics Explorer but selecting the location column:

Currently, Harness does not support multi-region load balancers. Add a Stackdriver step for each region.
  1. Select Log Verification or Metric Verification. See their sections below.

Logs Verification

  1. Configure the following settings.

Field

Description

Search Keywords

Enter search keywords for your query. You can use the same filters you have in GCP Logs Viewer.

Simply copy a filer entry into Search Keywords:

To use multiple filter entries, you can place an AND between them or use multiline entries. For example:

For advanced filter examples, see Advanced filters library from GCP.

Host Name Field

Enter the log field that contains the name of the host for which you want logs. You can enter a pod ID or name.

For example, the query in Search Keywords looks for pods labelled nginx-deployment:

resource.type="container"
resource.labels.pod_id:"nginx-deployment-"

In Host Name Field, you would enter pod_id because it is the log field containing the pod name. In a log, this field will be in the resource section:

...
 resource: {
  labels: {
   cluster_name:  "doc-example"    
   container_name:  "harness-delegate-instance"    
   instance_id:  "1733097732247470454"    
   namespace_id:  "harness-delegate"    
   pod_id:  "harness-sample-k8s-delegate-wverks-0"    
   project_id:  "exploration-161417"    
   zone:  "us-central1-a"    
  }
  type:  "container"   
 }
...

Algorithm Sensitivity

Select the Algorithm Sensitivity. See CV Strategies, Tuning, and Best Practices.

Enable 24/7 Service Guard

Click the checkbox to enable 24/7 Service Guard.

Baseline

Select the baseline time unit for monitoring. For example, if you select For 4 hours, Harness will collect the logs for the last 4 hours as the baseline for comparisons with future logs. If you select Custom Range you can enter a Start Time and End Time.

When you are finished, the dialog will look something like this:

  1. Click TEST to test your settings. In the testing assistant, select a host and click RUN. When you have confirmed you settings, click SUBMIT.
    1. The Stackdriver verification step is added to your Workflow.

Metrics Verification

  1. Configure the following settings.

Field

Description

Load Balancer Metrics

In Load Balancer, select the IP address of the load balancer to use. In Metric Names, select the GCP metrics for Harness to analyze. Currently, Harness analyzes CPU and memory metrics only.

You can find the IP address used by a load balancer on its Frontends tab:

Pod Metric Names

Select the CPU and memory metrics to analyze. These are the same metrics you use in Stackdriver Metrics Explorer:

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

See CV Strategies, Tuning, and Best Practices.

Baseline for Risk Analysis

Canary Analysis - Harness will compare the metrics received for the nodes deployed in each phase with metrics received for the rest of the nodes in the application. For example, if this phase deploys to 25% of your nodes, the metrics received from Stackdriver during this deployment for these nodes will be compared with metrics received for the other 75% during the defined period of time.

Previous Analysis - Harness will compare the metrics received for the nodes deployed in each phase with metrics received for all the nodes during the previous deployment. For example, if this phase deploys V1.2 to node A, the metrics received from Stackdriver during this deployment will be compared to the metrics for nodes A, B, and C during the previous deployment (V1.1).

See CV Strategies, Tuning, and Best Practices.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

When are done, the dialog will look something like this:

  1. Click TEST to test your settings. In the testing assistant, select a host and click RUN. The host name is the same as you can see in the Stackdriver Metrics Explorer:

If the settings are correct, and the Harness Delegate can reach the platform via its API, the assistant displays the data it found:

  1. When you are finished, click SUBMIT. The Stackdriver verification step is added to your Workflow.

Verification Results

Once you have deployed your Workflow (or Pipeline) using the Stackdriver verification step, you can automatically verify performance across your deployment. For more information, see Add a Workflow and Add a Pipeline.

Workflow Verification

To see the results of Harness machine-learning evaluation of your Stackdriver verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the Stackdriver step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The Workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

Next Steps

How did we do?