Stackdriver Verification

Updated 4 days ago by Michael Cretzman

The following sections describe how Harness integrates Stackdriver into Harness Continuous Verification to monitor your live, production services and verify your deployments:

Stackdriver and Harness

Google Stackdriver aggregates metrics, logs, and events from infrastructure, giving developers and operators a rich set of observable signals that speed root-cause analysis and reduce mean time to resolution (MTTR).

Harness Continuous Verification integrates with Stackdriver to verify your deployments and live production applications using the following Harness features:

  • 24/7 Service Guard- Monitors your live, production applications.
  • Deployment Verification- Monitors your application deployments, and performs automatic rollback according to your criteria.

This document describes how to set up these Harness Continuous Verification features and monitor your deployments and production applications using its unsupervised machine-learning functionality.

Verification with Stackdriver

Harness Analysis

You can read more about Harness and Stackdriver integration on the Harness Blog.

Setup Overview

You set up Stackdriver and Harness in the following way:

  1. Stackdriver - Monitor your application using Stackdriver. In this article, we assume that you are using Stackdriver to monitor your application already.
  2. Cloud Provider Setup - In Harness, you connect Harness to your Google account, adding Google Cloud Platform as a Harness Cloud Provider. For more informations, see Add Cloud Providers.
  3. Harness Application- Create a Harness Application with a Service, Environment, and Workflow. We do not cover Application set up in this article. See Application Components.
  4. ​24/7 Service Guard Setup - In the Environment, set up 24/7 Service Guard to monitor your live, production application.
  5. Verify Deployments:
    1. Add a Workflow to your Harness Application and deploy your microservice or application to the service infrastructure in your Environment.
    2. After you have run a successful deployment, you then add verification steps to the Workflow using your Verification Provider.
    3. Harness uses unsupervised machine-learning and Stackdriver monitoring to analyze your future deployments, discovering events that might be causing your deployments to fail. Then you can use this information to set rollback criteria and improve your deployments.

Verification Provider Setup

Connect Harness to Stackdriver to have Harness verify the success of your deployments. Harness will use your tools to verify deployments and use its machine learning features to identify sources of failures.

Most APM and logging tools are added to Harness as Verification Providers. For Stackdriver, you use the Google Cloud Platform account set up as a Harness Cloud Provider.

To add Stackdriver as a Cloud Provider, follow the steps for adding a Google Cloud Platform Cloud Provider.

Roles and Permissions

The minimum role requirement is monitoring.viewer. See Access control from Google.

The compute.networkViewer role will give read access to all networking resources and list forwarding rules.

24/7 Service Guard Setup

Harness 24/7 Service Guard monitors your live applications, catching problems that surface minutes or hours following deployment. For more information, see 24/7 Service Guard.

You can add your Stackdriver monitoring to Harness 24/7 Service Guard in your Harness Application Environment. For a setup overview, see Setup Preview.

For more information on 24/7 Service Guard, see 24/7 Service Guard.

To set up 24/7 Service Guard for Sumo Logic, do the following:

  1. Ensure that you have added Sumo Logic as a Harness Verification Provider, as described in Verification Provider Setup.
  2. In your Harness Application, ensure that you have added a Service, as described in Services. For 24/7 Service Guard, you do not need to add an Artifact Source to the Service, or configure its settings. You simply need to create a Service and name it. It will represent your application for 24/7 Service Guard.
  3. In your Harness Application, click Environments.
  4. In Environments, ensure that you have added an Environment for the Service you added. For steps on adding an Environment, see Environments.
  5. Click the Environment for your Service. Typically, the Environment Type is Production.
  6. In the Environment page, locate 24/7 Service Guard.
  7. In 24/7 Service Guard, click Add Service Verification, and then click Stackdriver. The Stackdriver dialog appears.

  1. Fill out the dialog. The dialog has the following fields. For information on the log entries used, see Viewing Logs from GCP.
For 24/7 Service Guard, the queries you define to collect logs are specific to the application or service you want monitored. Verification is application/service level. This is unlike Workflows, where verification is performed at the host/node/pod level.

Field

Description

Display Name

The name that will identify this service on the Continuous Verification dashboard. Use a name that indicates the environment and monitoring tool, such as Stackdriver.

Service

The Harness Service to monitor with 24/7 Service Guard.

GCP Cloud Provider

Select the GCP Cloud Provider to use, as described in Verification Provider Setup. If you current connect to GCP via a Kubernetes Cluster Cloud Provider, you must set up a GCP Cloud Provider for access to the Stackdriver data on your cluster.

Search Keywords

Enter search keywords for your query. You can use the same filters you have in GCP Logs Viewer.

Simply copy a filer entry into Search Keywords:

To use multiple filter entries, place an AND between them. For example:

For advanced filter examples, see Advanced filters library from GCP.

Host Name Field

Enter the log field that contains the name of the host/pod/container for which you want logs. You can enter a pod ID or field name for example.

Harness uses this field to group data and perform analysis at the container-level.

For example, the query in Search Keywords looks for pods labelled nginx-deployment:

resource.type="container"
resource.labels.pod_id:"nginx-deployment-"

In Host Name Field, you would enter pod_id because it is the log field containing the pod name. In a log, this field will be in the resource section:

...
resource: {
labels: {
cluster_name: "doc-example"
container_name: "harness-delegate-instance"
instance_id: "1733097732247470454"
namespace_id: "harness-delegate"
pod_id: "harness-sample-k8s-delegate-wverks-0"
project_id: "exploration-161417"
zone: "us-central1-a"
}
type: "container"
}
...

Algorithm Sensitivity

Select the Algorithm Sensitivity.

Enable 24/7 Service Guard

Click the checkbox to enable 24/7 Service Guard.

Baseline

Select the baseline time unit for monitoring. For example, if you select For 4 hours, Harness will collect the logs for the last 4 hours as the baseline for comparisons with future logs. If you select Custom Range you can enter a Start Time and End Time.

When you are finished, the dialog will look something like this:

  1. Click TEST. Harness verifies the settings you entered.
  2. Click SUBMIT. The Stackdriver 24/7 Service Guard is configured.

To see the running 24/7 Service Guard analysis, click Continuous Verification. The 24/7 Service Guard dashboard displays the production verification results.

Verify Deployments

Harness can analyze Stackdriver data and analysis to verify, rollback, and improve deployments. To apply this analysis to your deployments, you set up Stackdriver as a verification step in a Harness Workflow.

This section covers how to set up Stackdriver in a Harness Workflow, and provides a summary of Harness verification results.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

You can use both Stackdriver Logs and Metrics in a Workflow.

To verify your deployment with Stackdriver, do the following:

  1. Ensure that you have added Google Cloud Platform as a Cloud Provider provider, as described above.
  2. In your Workflow, under Verify Service, click Add Verification, and then click Stackdriver. The Stackdriver dialog appears.

  1. In GCP Cloud Provider, select the Google Cloud Platform (GCP) Cloud Provider you set up in Harness.
  2. In Region, select the GCP region where the application is hosted. The Stackdriver API uses a service-specific notion of location. Harness uses the name of a region. You can find the region in Stackdriver Metrics Explorer but selecting the location column:

Currently, Harness does not support multi-region load balancers. Add a Stackdriver step for each region.
  1. Select Log Verification or Metric Verification. See their sections below.

Logs Verification

  1. Configure the following settings.

Field

Description

Search Keywords

Enter search keywords for your query. You can use the same filters you have in GCP Logs Viewer.

Simply copy a filer entry into Search Keywords:

To use multiple filter entries, place an AND between them. For example:

For advanced filter examples, see Advanced filters library from GCP.

Host Name Field

Enter the log field that contains the name of the host for which you want logs. You can enter a pod ID or name.

For example, the query in Search Keywords looks for pods labelled nginx-deployment:

resource.type="container"
resource.labels.pod_id:"nginx-deployment-"

In Host Name Field, you would enter pod_id because it is the log field containing the pod name. In a log, this field will be in the resource section:

...
resource: {
labels: {
cluster_name: "doc-example"
container_name: "harness-delegate-instance"
instance_id: "1733097732247470454"
namespace_id: "harness-delegate"
pod_id: "harness-sample-k8s-delegate-wverks-0"
project_id: "exploration-161417"
zone: "us-central1-a"
}
type: "container"
}
...

Algorithm Sensitivity

Select the Algorithm Sensitivity.

Enable 24/7 Service Guard

Click the checkbox to enable 24/7 Service Guard.

Baseline

Select the baseline time unit for monitoring. For example, if you select For 4 hours, Harness will collect the logs for the last 4 hours as the baseline for comparisons with future logs. If you select Custom Range you can enter a Start Time and End Time.

When you are finished, the dialog will look something like this:

  1. Click TEST to test your settings. In the testing assistant, select a host and click RUN. When you have confirmed you settings, click SUBMIT.
    1. The Stackdriver verification step is added to your Workflow.

Metrics Verification

  1. Configure the following settings.

Field

Description

Load Balancer Metrics

In Load Balancer, select the IP address of the load balancer to use. In Metric Names, select the GCP metrics for Harness to analyze. Currently, Harness analyzes CPU and memory metrics only.

You can find the IP address used by a load balancer on its Frontends tab:

Pod Metric Names

Select the CPU and memory metrics to analyze. These are the same metrics you use in Stackdriver Metrics Explorer:

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline for Risk Analysis

Select Previous Analysis to have this verification use the previous analysis for a baseline comparison. If your workflow is a Canary workflow type, you can select Canary Analysis to have this verification compare old versions of nodes to new versions of nodes in real-time.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

When are done, the dialog will look something like this:

  1. Click TEST to test your settings. In the testing assistant, select a host and click RUN. The host name is the same as you can see in the Stackdriver Metrics Explorer:

If the settings are correct, and the Harness Delegate can reach the platform via its API, the assistant displays the data it found:

  1. When you are finished, click SUBMIT. The Stackdriver verification step is added to your Workflow.

Verification Results

Once you have deployed your Workflow (or Pipeline) using the Stackdriver verification step, you can automatically verify performance across your deployment. For more information, see Add a Workflow and Add a Pipeline.

Workflow Verification

To see the results of Harness machine-learning evaluation of your Stackdriver verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the Stackdriver step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The Workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

Next Steps


How did we do?