2 - Verify Deployments with Custom APMs

Updated 2 weeks ago by Michael Cretzman

The following procedure describes how to add a custom APM (metrics) or Logs verification step in a Harness Workflow. For more information about Workflows, see Add a Workflow.

Once you run a deployment and your custom metrics or logs provider obtains its data, Harness machine-learning verification analysis will assess the risk level of the deployment using the data from the provider.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your Workflow after you have run at least one successful deployment.

Deployment Verification Setup

To verify your deployment with a custom metric or log provider, do the following:

  1. Ensure that you have added Custom Metrics or Logs Provider as a verification provider, as described in Connect to Custom Provider.
  2. In your workflow, under Verify Service, click Add Verification.
  3. In the Add Command popover, click APM Verification or Log Verification. The Metrics Verification State or Custom Logs Verification dialog appears.

As you can see, the dialogs are the same except for the Metric Collections Settings and Log Collection Settings sections. For other, common settings, see Common Settings.

Metric Collections Settings

  1. In Metrics Data Provider, select the custom metric provider you added, described in Connect to Custom Provider.
  2. Add a Metric Collections section.

Metric Collections is covered in the following table.

Field

Description

Metrics Name

Enter a name for the type of error you are collecting, such as HttpErrors.

Metrics Type

Select the type of metric you want to collect:

  • Infra - Infrastructure metrics, such as CPU, memory, and HTTP errors.
  • Value - Apdex (measures user satisfaction with response time).
  • Response Time - The amount of time the application takes to return a request.
  • Throughput - The average number of units processed per time unit.
  • Error - Errors reported.

Metrics Group

Enter the name to use in the Harness UI to identify this metrics group. For example, if you entered NRHttp in this field, when the workflow is deployed, the verification output in Harness would look like this:

Metrics Collection URL

Enter a query for your verification. You can simply make the query in your Verification Provider and paste it in this field. For example, in New Relic Insights, you might have the following query:

You can paste the query into the Metrics Collection URL field:

For information on New Relic Insights NRSQL, see NRQL syntax, components, functions from New Relic.The time range for a query (SINCE clause in our example) should be less than 5 minutes to avoid overstepping the time limit for some verification providers.

Response Type

Select the format type for the response. This is typically JSON.

Metrics Method

Select GET or POST. If you select POST, the Metric Collection Body field appears.

In Metric Collection Body, enter the JSON body that to send as a payload when making a REST call to the APM Provider. The requirements of the JSON body will depend on your APM provider.

You can use variables you created in the Service and Workflow in the JSON, as well as Harness built-in variables.

Response Mapping

These settings are for specifying which JSON fields in the responses to use.

Transaction Name

Fixed: Use this option when all metrics are for the same transaction. For example, a single login page.Dynamic: Use this option when the metrics are for multiple transactions.

Transaction Name Path

This is the JSON label for identifying a transaction name. In the case of our example New Relic Insights query, the FACET clause is used to group results by the attribute transactionName. You can obtain the field name that rcords the transactionName by using the Guide From Example feature:

  1. Click Guide From Example. The Select key from example popover appears.
    The Metrics URL Collection is based on the query you entered in the Metric Collection URL field earlier.
  2. Click Submit. The query is executed and the JSON is returned.
  3. Locate the field name that is used to identify transactions. In our New Relic Insights query, it is the facets.name field.
    In New Relic Insights, you can find the name in the JSON of your query results.
  4. Click the field name under facets. The field path is added to the Transaction Name Path field.

Metrics Value

Specify the value for the event count. This is used to filter and aggregate data returned in a SELECT statement. To find the correct label for the value, do the following:

  1. Click Guide From Example. The Select key from example popover appears.
    The Metrics URL Collection is based on the query you entered in the Metric Collection URL field earlier.
  2. Click Submit. The query is executed and the JSON is returned.
  3. Locate the field name that is used to count events. In our New Relic Insights query, it is the facets.timeSeries.results.count field.
    In New Relic Insights, you can find the name in the JSON of your query results.
  4. Click the name of the field count. The field path is added to the Metrics Value field.

Timestamp

Specify the value for the timestamp in the query. To find the correct label for the value, do the following:

  1. Click Guide From Example. The Select key from example popover appears.
    The Metrics URL Collection is based on the query you entered in the Metric Collection URL field earlier.
  2. Click Submit. The query is executed and the JSON is returned.
  3. Locate the field name that is used for the time series endTimeSeconds. In our New Relic Insights query, it is the facets.timeSeries.endTimeSeconds field.
    In New Relic Insights, you can find the name in the JSON of your query results.
  4. Click the name of the field endTimeSeconds. The field path is added to the Timestamp field.

Timestamp Format

Enter a timestamp format. The format follows the Java SimpleDateFormat. For example, a timestamp syntax might be yyyy-MM-dd'T'HH:mm:ss.SSSX. If you leave this field empty, Harness will use the default range of 1 hour previous (now-1h) to now.

When you are done, the dialog will look something like this:

Log Collection Settings

  1. In Logs Data Provider, select the custom logs provider you added, described in Connect to Custom Provider.
  2. Click Add Collection Details to add a Logs Collections section.

  • In Log Collection URL, enter the API query that will return a JSON response. When you enter a query, the Response Mapping settings appear. In this section you will map the keys in the JSON response to Harness fields to identify where data such as the hostname and timestamp are located in the JSON response.

  • In Response Type,- Select JSON.
  • In Metrics Method - Select GET or POST. If you select POST, you also enable a Log Collection Body field where you can enter any HTTP body to send as part of the query.
  • In Log Message JSON Path - Use Guide from an example to query the log provider and return the JSON response.

The URL is a combination of the Verification Cloud Provider Base URL and the Log Collection URL you entered.

Click SEND. In the JSON response, click the key that includes the log message path.

The log message path key is added to Log Message JSON Path:

  • Hostname JSON Path - Use Guide from an example to query the log provider and return the JSON response. In the JSON response, click the key that includes the hostname path.

  • Regex to transform Host Name - If the JSON value returned requires transformation in order to be used, enter the regex expression here. For example: If the value in the host name JSON path of the response is pod_name:harness-test.pod.name and the actual pod name is simply harness-test.pod.name, you can write a regular expression to remove the pod_name from the response value.
  • Timestamp JSON path - Use Guide from an example to query the log provider and return the JSON response. In the JSON response, click the key that includes the timestamp.

  • Timestamp Format - Enter a timestamp format. The format follows the Java SimpleDateFormat. For example, a timestamp syntax might be yyyy-MM-dd'T'HH:mm:ss.SSSX. If you leave this field empty, Harness will use the default range of 1 hour previous (now - 1h) to now.

Click ADD. The Log Collection is added.

Common Settings

The following settings are common to both dialogs.

Field

Description

Expression for Host/Container

The expression entered here should resolve to a host/container name in your deployment environment. By default, the expression is ${host.hostName}. If you begin typing the expression into the field, the field provides expression assistance.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Harness waits 2-3 minutes before beginning the analysis to avoid initial deployment noise. This is a standard with monitoring tools.

Baseline for Risk Analysis

Canary Analysis - Harness will compare the metrics received for the nodes deployed in each phase with metrics received for the rest of the nodes in the application. For example, if this phase deploys to 25% of your nodes, the metrics received from Custom APM during this deployment for these nodes will be compared with metrics received for the other 75% during the defined period of time.

Previous Analysis - Harness will compare the metrics received for the nodes deployed in each phase with metrics received for all the nodes during the previous deployment. For example, if this phase deploys V1.2 to node A, the metrics received from Custom APM during this deployment will be compared to the metrics for nodes A, B, and C during the previous deployment (V1.1). Previous Analysis is best used when you have predictable load, such as in a QA environment.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

Include instances from previous phases

If you are using this verification step in a multi-phase deployment, select this checkbox to include instances used in previous phases when collecting data. Do not apply this setting to the first phase in a multi-phase deployment.

Wait interval before execution

Set how long the deployment process should wait before executing the verification step.

Notes

  • Depending on the custom metric provider you select, you might need to provide different information to the Metric Collections section. For example, you might need to provide a hostname for the Guide From Example popover to use to retrieve data. The hostname will be the host/container/pod/node name where the artifact is deployed. In you look in the JSON for the deployment environment, the hostname is typically the name label under the host label.
  • The Compare With Previous Run option is used for Canary deployments where the second phase is compared to the first phase, and the third phase is compared to the second phase, and so on. Do not use this setting in a single phase workflow or in the first phase of a multi-phase workflow.

Next Step


How did we do?