Datadog Verification

Updated 3 days ago by Michael Cretzman

The following sections describe how Harness integrates ​Datadog into Harness Continuous Verification to monitor your live, production services and verify your deployments:

Datadog and Harness

Datadog delivers real-time and trending data about application performance by seamlessly aggregating metrics and events across the full DevOps stack. Datadog automatically collects logs from all your services, applications, and platforms.

Harness Continuous Verification integrates with Datadog to verify your deployments and live production applications using the following Harness features:

  • 24/7 Service Guard - Monitors your live, production applications.
  • Deployment Verification - Monitors your application deployments, and performs automatic rollback according to your criteria.
At this time, Datadog Deployment Verification is supported for Harness Kubernetes Service deployments only. To add deployment verification in your non-Kubernetes Workflow, use Custom Metrics Verification and your Datadog monitoring. Datadog is fully supported for 24/7 Service Guard.

This document describes how to set up these Harness Continuous Verification features and monitor your deployments and production applications using its unsupervised machine-learning functionality.

Analysis with Datadog

Harness Analysis

Setup Preview

You set up Datadog and Harness in the following way:

  1. Datadog - Monitor your application using Datadog. In this article, we assume that you are using Datadog to monitor your application already.
  2. ​Verification Provider Setup - In Harness, you connect Harness to your Datadog account, adding Datadog as a Harness Verification Provider.
  3. Harness Application - Create a Harness Application with a Service and an Environment. We do not cover Application set up in this article. See Application Checklist.
  4. ​24/7 Service Guard Setup- In the Environment, set up 24/7 Service Guard to monitor your live, production application.
  5. Verify Deployments:
    1. Add a Workflow to your Harness Application and deploy your microservice or application to the service infrastructure in your Environment.
    2. After you have run a successful deployment, you then add verification steps to the Workflow using your Verification Provider.
    3. Harness uses unsupervised machine-learning and Datadog analytics to analyze your future deployments, discovering events that might be causing your deployments to fail. Then you can use this information to set rollback criteria and improve your deployments.

Verification Provider Setup

The first step in using Datadog with Harness is to set up an Datadog Verification Provider in Harness.

A Harness Verification Provider is a connection to monitoring tools such as Datadog. Once Harness is connected, you can use Harness 24/7 Service Guard and Deployment Verification with your Datadog data and analysis.

To add Datadog as a verification provider, do the following:

  1. Click Setup.
  2. Click Connectors, and then click Verification Providers.
  3. Click Add Verification Provider, and select Datadog. The Datadog dialog for your provider appears.

  1. Complete the following fields of the Add Datadog Verification Provider dialog.
You need Datadog Admin access to create the API key needed to connect Harness to Datadog.

Field

Description

URL

Enter the URL of the Datadog server. Simply take the URL from the Datadog dashboard, such as https://app.datadoghq.com/ and add the API and version: https://app.datadoghq.com/api/v1/.

API Key

Enter the API key for API calls.

To create an API key in Datadog, do the following:

  1. In Datadog, mouseover Integrations, and then click APIs.
    The APIs page appears.
  2. In API Keys, in New API key, enter the name for the new API key, such as Harness, and then click Create API key.
  3. Copy the API key and, in Harness, paste it into the API Key field.

Application Key

Enter the application key.

To create an application key in Datadog, do the following:

  1. In Datadog, mouseover Integrations, and then click APIs. The APIs page appears.
  2. In Application Keys, in New application key, enter a name for the application key, such as Harness, and click Create Application Key.
  3. Copy the API key and, in Harness, paste it into the Application Key field.

Display Name

Enter a display name for the provider. If you are going to use multiple providers of the same type, ensure you give each provider a different name.

Usage Scope

If you want to restrict the use of a provider to specific applications and environments, do the following:

In Usage Scope, click the drop-down under Applications, and click the name of the application.

In Environments, click the name of the environment.

Datadog has limit of about 300 API calls per hour. Requests to analyze many metrics can hit the limit. Datadog can increase the limit upon request. For more information, see Rate Limiting from Datadog.

24/7 Service Guard Setup

Harness 24/7 Service Guard monitors your live applications, catching problems that surface minutes or hours following deployment. For more information, see 24/7 Service Guard.

You can add your Datadog monitoring to Harness 24/7 Service Guard in your Harness Application Environment. For a setup overview, see Setup Preview.

This section assumes you have a Harness Application set up and containing a Service and Environment. For steps on setting up a Harness Application, see Application Checklist.

To set up 24/7 Service Guard for Datadog, do the following:

  1. Ensure that you have added Datadog as a Harness Verification Provider, as described in Verification Provider Setup.
  2. In your Harness Application, ensure that you have added a Service, as described in Services. For 24/7 Service Guard, you do not need to add an Artifact Source to the Service, or configure its settings. You simply need to create a Service and name it. It will represent your application for 24/7 Service Guard.
  3. In your Harness Application, click Environments.
  4. In Environments, ensure that you have added an Environment for the Service you added. For steps on adding an Environment, see Environments.
  5. Click the Environment for your Service. Typically, the Environment Type is Production.
  6. In the Environment page, locate 24/7 Service Guard.
  7. In 24/7 Service Guard, click Add Service Verification, and then click Datadog. The Datadog dialog appears.
  8. Fill out the dialog. The dialog has the following fields.
For 24/7 Service Guard, the queries you define to collect logs are specific to the application or service you want monitored. Verification is application/service level. This is unlike Workflows, where verification is performed at the host/node/pod level.

Field

Description

Display Name

The name that will identify this service on the Continuous Verification dashboard. Use a name that indicates the environment and monitoring tool, such as Datadog.

Service

The Harness Service to monitor with 24/7 Service Guard.

Datadog Server

Select the Datadog Verification Provider to use.

APM Traces

In Datadog Service Name, enter which service in Datadog you want to monitor. By default, Datadog monitors the servlet hits (number of incoming requests), servlet duration, and errors. Harness will fetch this data for every Web transaction with the service you enter and display it in 24/7 Service Guard as Errors, Hits, Request Duration.

Docker Containers

Select the Docker container infrastructure metrics to monitor.

In Datadog Tags, enter any tags that have been applied to your metrics in Datadog. These are the same tags used in Datadog Events, Metrics Explorer, etc.

Use the Datadog tag format, such as cluster-name:harness-test.

Failure Criteria

Specify the sensitivity of the failure criteria.

Enable 24/7 Service Guard

Enable this setting to turn on 24/7 Service Guard. If you simply want to set up 24/7 Service Guard, but not enable it, leave this setting disabled.

When you are finished, the dialog will look something like this:

  1. Click TEST. Harness verifies the settings you entered.
  2. Click SUBMIT. The Datadog 24/7 Service Guard to configured.

To see the running 24/7 Service Guard analysis, click Continuous Verification.

The 24/7 Service Guard dashboard displays the production verification results.

For information on using the dashboard, see Using 24/7 Service Guard.

Verify Deployments

Harness can analyze Datadog data and analysis to verify, rollback, and improve deployments. To apply this analysis to your deployments, you set up Datadog as a verification step in a Harness Workflow.

Once you run a deployment and Datadog preforms verification, Harness' machine-learning verification analysis will assess the risk level of the deployment.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

Deployment Verification Setup

To verify your deployment with Datadog, do the following:

  1. Ensure that you have added Datadog as a verification provider, as described above.
  2. In your workflow, under Verify Service, click Add Verification, and then click Datadog. The Datadog dialog appears.

  1. The Datadog dialog has the following fields.

Field

Description

Datadog Server

Select the Datadog verification provider you added, as described above.

Metrics

Select any of the Datadog API metrics. For a list of the API metrics, see Data Collected from Datadog.

Expression for Host/Container name

Enter an expression that evaluates to the host/container/pod name tagged in the Datadog events.

For example, in Datadog, a Kubernetes deployment might use the tag pod_name to identify the pod where the microservice is deployed. Find the where the same name is identified in the deployment environment, and use that path as the expression.

For example, locate the pod name in the Datadog Event Stream page:

  1. In Datadog, click Events.
  2. Locate an event using a search query. For more information, see Event Stream from Datadog.
  3. Expand the event by click the the ellipsis at the end of the event title.
  4. Look through the event details and locate the tag that lists the pod name for the instance where the service is deployed. In our example, the tag is pod_name.
  5. Next, look in the JSON for the host/container/pod in the deployment environment and identify the label containing the same hostname. The path to that label is what the expression should be in Expression for Host/Container name. The default expression is ${host.hostName}. In most cases, this expression will work.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline for Risk Analysis

Select Previous Analysis to have this verification use the previous analysis for a baseline comparison. If your workflow is a Canary workflow type, you can select Canary Analysis to have this verification compare old versions of nodes to new versions of nodes in real-time.

For Canary Analysis and Previous Analysis, analysis happens at the host/node/pod level.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

  1. Click TEST. Harness verifies the settings you entered.
  2. When you are finished, click SUBMIT. The Datadog verification step is added to your workflow.

Verification Results

Once you have deployed your workflow (or pipeline) using the Datadog verification step, you can automatically verify cloud application and infrastructure performance across your deployment. For more information, see Add a Workflow and Add a Pipeline.

Workflow Verification

To see the results of Harness machine-learning evaluation of your Datadog verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the Datadog step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

To learn about the verification analysis features, see the following sections.

Deployments

Deployment info
See the verification analysis for each deployment, with information on its service, environment, pipeline, and workflows.

Verification phases and providers
See the vertfication phases for each vertfication provider. Click each provider for logs and analysis.

Verification timeline
See when each deployment and verification was performed.

Transaction Analysis

Execution details
See the details of verification execution. Total is the total time the verification step took, and Analysis duration is how long the analysis took.

Risk level analysis
Get an overall risk level and view the cluster chart to see events.

Transaction-level summary
See a summary of each transaction with the query string, error values comparison, and a risk analysis summary.

Execution Analysis

Event type
Filter cluster chart events by Unknown Event, Unexpected Frequency, Anticipated Event, Baseline Event, and Ignore Event.

Cluster chart
View the chart to see how the selected event contrast. Click each event to see its log details.

Event Management

Event-level analysis
See the threat level for each event captured.

Tune event capture
Remove events from analysis at the service, workflow, execution, or overall level.

Event distribution
Click the chart icon to see an event distribution including the measured data, baseline data, and event frequency.

Next Steps


How did we do?