Datadog Verification

Updated 4 months ago by Michael Cretzman

Datadog delivers real-time and trending data about application performance by seamlessly aggregating metrics and events across the full DevOps stack. Datadog automatically collects logs from all your services, applications, and platforms.

You can add a Datadog verification step to your workflow and Datadog will be used by Harness to verify the performance and quality of your deployments using Harness machine-learning verification analysis.

Analysis with Datadog

Harness Analysis

Verification Setup Overview

You set up Datadog and Harness in the following way:

  1. Using Datadog, you monitor your microservice or application.
  2. In Harness, you connect Harness to your Datadog account, adding Datadog as a Harness Verification Provider.
  3. After you have built and run a successful deployment of your microservice or application in Harness, you then add Datadog verification steps to your Harness deployment workflow. You add Datadog after a successful deployment in order that Harness can use Datadog on the specific hosts/containers/pods on which the microservice is deployed, using the deployment environment tags or labels that identify the hosts/containers/pods.
  4. Harness uses Datadog to verify your future microservice/application deployments.
  5. Harness Continuous Verification uses unsupervised machine-learning to analyze your deployments and Datadog analytics/logs, discovering events that might be causing your deployments to fail. Then you can use this information to improve your deployments.

Intended Audience

  • Developers
  • DevOps

Before You Begin

Connect to Datadog

Connect Harness to Datadog to have Harness verify the success of your deployments. Harness will use your tools to verify deployments and use its machine learning features to identify sources of failures.

To add Datadog as a verification provider, do the following:

  1. Click Setup.
  2. Click Connectors.
  3. Click Verification Providers.
  4. Click Add Verification Provider, and select Datadog. The Datadog dialog for your provider appears.

The Add Datadog Verification Provider dialog has the following fields.

You need Datadog Admin access to create the API key needed to connect Harness to Datadog.

Field

Description

URL

Enter the URL of the Datadog server. Simply take the URL from the Datadog dashboard, such as https://app.datadoghq.com/ and add the API and version: https://app.datadoghq.com/api/v1/.

API Key

Enter the API key for API calls.

To create an API key in Datadog, do the following:

  1. In Datadog, mouseover Integrations, and then click APIs.


    The APIs page appears.
  2. In API Keys, in New API key, enter the name for the new API key, such as Harness, and then click Create API key.
  3. Copy the API key and, in Harness, paste it into the API Key field.

Application Key

Enter the application key.

To create an application key in Datadog, do the following:

  1. In Datadog, mouseover Integrations, and then click APIs. The APIs page appears.
  2. In Application Keys, in New application key, enter a name for the application key, such as Harness, and click Create Application Key.
  3. Copy the API key and, in Harness, paste it into the Application Key field.

Display Name

Enter a display name for the provider. If you are going to use multiple providers of the same type, ensure you give each provider a different name.

Usage Scope

If you want to restrict the use of a provider to specific applications and environments, do the following:

In Usage Scope, click the drop-down under Applications, and click the name of the application.

In Environments, click the name of the environment.

Datadog has limit of about 300 API calls per hour. Requests to analyze many metrics can hit the limit. Datadog can increase the limit upon request. For more information, see Rate Limiting from Datadog.

Verify with Datadog

The following procedure describes how to add Datadog as a verification step in a Harness workflow. For more information about workflows, see Add a Workflow.

Once you run a deployment and Datadog preforms verification, Harness' machine-learning verification analysis will assess the risk level of the deployment.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

To verify your deployment with Datadog, do the following:

  1. Ensure that you have added Datadog as a verification provider, as described above.
  2. In your workflow, under Verify Service, click Add Verification, and then click Datadog. The Datadog dialog appears.

The Datadog dialog has the following fields.

Field

Description

Datadog Server

Select the Datadog verification provider you added, as described above.

Datadog Service Name

For infrastructure and system-level metrics, you can leave this field empty. For any service-specific metrics, enter the service name that is emitting metrics to Datadog.

Metrics

Select any of the Datadog API metrics. For a list of the API metrics, see Data Collected from Datadog.

Expression for Host/Container name

Enter an expression that evaluates to the host/container/pod name tagged in the Datadog events.

For example, in Datadog, a Kubernetes deployment might use the tag pod_name to identify the pod where the microservice is deployed. Find the where the same name is identified in the deployment environment, and use that path as the expression.

For example, locate the pod name in the Datadog Event Stream page:

  1. In Datadog, click Events.
  2. Locate an event using a search query. For more information, see Event Stream from Datadog.
  3. Expand the event by click the the ellipsis at the end of the event title.
  4. Look through the event details and locate the tag that lists the pod name for the instance where the service is deployed. In our example, the tag is pod_name.
  5. Next, look in the JSON for the host/container/pod in the deployment environment and identify the label containing the same hostname. The path to that label is what the expression should be in Expression for Host/Container name. The default expression is ${host.hostName}. In most cases, this expression will work.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline for Risk Analysis

Select Previous Analysis to have this verification use the previous analysis for a baseline comparison. If your workflow is a Canary workflow type, you can select Canary Analysis to have this verification compare old versions of nodes to new versions of nodes in real-time.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

When you are finished, click SUBMIT. The Datadog verification step is added to your workflow.

Verification Results

Once you have deployed your workflow (or pipeline) using the Datadog verification step, you can automatically verify cloud application and infrastructure performance across your deployment. For more information, see Add a Workflow and Add a Pipeline.

Workflow Verification

To see the results of Harness machine-learning evaluation of your Datadog verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the Datadog step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

To learn about the verification analysis features, see the following sections.

Deployments

Deployment info
See the verification analysis for each deployment, with information on its service, environment, pipeline, and workflows.

Verification phases and providers
See the vertfication phases for each vertfication provider. Click each provider for logs and analysis.

Verification timeline
See when each deployment and verification was performed.

Transaction Analysis

Execution details
See the details of verification execution. Total is the total time the verification step took, and Analysis duration is how long the analysis took.

Risk level analysis
Get an overall risk level and view the cluster chart to see events.

Transaction-level summary
See a summary of each transaction with the query string, error values comparison, and a risk analysis summary.

Execution Analysis

Event type
Filter cluster chart events by Unknown Event, Unexpected Frequency, Anticipated Event, Baseline Event, and Ignore Event.

Cluster chart
View the chart to see how the selected event contrast. Click each event to see its log details.

Event Management

Event-level analysis
See the threat level for each event captured.

Tune event capture
Remove events from analysis at the service, workflow, execution, or overall level.

Event distribution
Click the chart icon to see an event distribution including the measured data, baseline data, and event frequency.

Next Steps


How did we do?