4 - Verify Deployments with Custom APMs and Logs

Updated 3 weeks ago by Michael Cretzman

The following procedure describes how to add a custom APM (metrics) or Logs verification step in a Harness Workflow. For more information about Workflows, see Add a Workflow.

Once you run a deployment and your custom metrics or logs provider obtains its data, Harness machine-learning verification analysis will assess the risk level of the deployment using the data from the provider.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your Workflow after you have run at least one successful deployment.

Deployment Verification Setup

To verify your deployment with a custom metric or log provider, do the following:

  1. Ensure that you have added Custom Metrics or Logs Provider as a verification provider, as described in 1 – Custom Verification Connection Setup.
  2. In your workflow, under Verify Service, click Add Verification.
  3. In the resulting Add Step settings, select either Performance Monitoring > Custom Metrics or Log Analysis > Custom Log Verification.
  4. Click Next. The Configure Custom Metrics or Configure Custom Logs Verification settings appear.

As you can see, the settings are the same except for the Metric Collections Settings and Log Collections Settings sections. For other, common settings, see Common Settings.

Metric Collections Settings

  1. In Metrics Data Provider, select the custom metric provider you added, described in 1 - Custom Verification Connection Setup.
  2. In Metrics Type, select Infrastructure or Transaction.
  3. Add a Metric Collections section.

Metric Collections are covered in the following table.

Field

Description

Metrics Name

Enter a name for the type of error you are collecting, such as HttpErrors.

Metrics Type

For the Infrastructure Metrics Type, select the type of metric you want to collect:

  • Infra -–Infrastructure metrics, such as CPU, memory, and HTTP errors.
  • ValueApdex (measures user satisfaction with response time).
  • Lower Value – Values below the average.

For the Transaction Metrics Type, select the type of metric you want to collect:

  • Error
  • Response Time
  • Throughput

Metrics Collection URL

Enter a query for your verification. You can simply make the query in your Verification Provider and paste it in this field. For example, in New Relic Insights, you might have the following query:

You can paste the query into the Metrics Collection URL field:

For information on New Relic Insights NRSQL, see NRQL syntax, components, functions from New Relic.The time range for a query (SINCE clause in our example) should be less than 5 minutes to avoid overstepping the time limit for some verification providers.

Most often, when you create your query, you will include a hostname placeholder in the query, ${host}. This placeholder will be used later when setting up the Metrics Value and other settings that use Guide from an example.

For example, if your query is:

SELECT count(host) FROM Transaction SINCE 30 MINUTES AGO COMPARE WITH 1 WEEK AGO WHERE host = '37c444347ac2' TIMESERIES

Then you replace the host name with the ${host} placeholder and paste the query into Metrics Collection URL:

SELECT count(host) FROM Transaction SINCE 30 MINUTES AGO COMPARE WITH 1 WEEK AGO WHERE host = '${host}' TIMESERIES

Metrics Method

Select GET or POST. If you select POST, the Metric Collection Body field appears.

In Metric Collection Body, enter the JSON body to send as a payload when making a REST call to the APM Provider. The requirements of the JSON body will depend on your APM provider.

You can use variables you created in the Service and Workflow in the JSON, as well as Harness built-in variables.

Response Mapping – Transaction Name

These settings are for specifying which JSON fields in the responses to use.

Select Fixed or Dynamic.

Fixed: Use this option when all metrics are for the same transaction. For example, a single login page.

Dynamic: Use this option when the metrics are for multiple transactions.

Name (Fixed)

Enter the name with which you want to identify the transaction.

Transaction Name Path (Dynamic)

This is the JSON label for identifying a transaction name. In the case of our example New Relic Insights query, the FACET clause is used to group results by the attribute transactionName. You can obtain the field name that records the transactionName by using the Guide from an example feature:

  1. Click Guide from an example. The Select Key from Example popover appears.
    The Metrics URL Collection is based on the query you entered in the Metric Collection URL field earlier.
  2. In ${host}, select a host to query. Click the query next to GET to see how the host you selected replaces the ${host} placeholder in your query.
  3. Click SEND. The query is executed and the JSON is returned.
  4. Locate the field name that is used to identify transactions. In our New Relic Insights query, it is the facets.name field.
    If no metrics are found, you will see a METRIC DATA NOT FOUND error.
    In New Relic Insights, you can find the name in the JSON of your query results.
  5. Click the field name under facets. The field path is added to the Transaction Name Path field.

Regex to transform Transaction Name (Dynamic)

Enter a regex expression here to obtain the specific name from the transaction path.

For example, if your Transaction Name Path JSON evaluated to a value such as name : TxnName, you can write a regex to remove everything other than TxnName.

For example (name:(.*),) or (?<=:).*(?=,).

Metrics Value

Specify the value for the event count. This is used to filter and aggregate data returned in a SELECT statement. To find the correct label for the value, do the following:

  1. Click Guide from an example. The example popover appears.
    The Metrics URL Collection is based on the query you entered in the Metric Collection URL field earlier. The ${host} field refers to the ${host} variable in your query.
  2. Click Submit. The query is executed and the JSON is returned.
    If no metrics are found, you will see a METRIC DATA NOT FOUND error.
  3. Locate the field name that is used to count events. In our New Relic Insights query, it is the facets.timeSeries.results.count field.
    In New Relic Insights, you can find the name in the JSON of your query results.
  4. Click the name of the field count. The field path is added to the Metrics Value field.

Hostname JSON path

(Displayed if ${host} is present in the Metrics Collection URL query)

Use Guide from an example to select a host and query your APM. Click the name of the hostname JSON label in the response.

If there is no hostname in the response, leave this setting empty.

Timestamp

Specify the value for the timestamp in the query. To find the correct label for the value, do the following:

  1. Click Guide from an example. The Select Key from Example popover appears.
    The Metrics URL Collection is based on the query you entered in the Metric Collection URL field earlier.
  2. Click Submit. The query is executed and the JSON is returned.
  3. Locate the field name that is used for the time series endTimeSeconds. In our New Relic Insights query, it is the facets.timeSeries.endTimeSeconds field.
    In New Relic Insights, you can find the name in the JSON of your query results.
  4. Click the name of the field endTimeSeconds. The field path is added to the Timestamp field.

Timestamp Format

Enter the format of the timestamp included in the query request (not response), set in Timestamp. The format follows the Java SimpleDateFormat. For example, a timestamp syntax might be yyyy-MM-dd'T'HH:mm:ss.SSSX. If you leave this field empty, Harness will use the default range of 1 hour previous (now-1h) to now.

When you are done, the settings will look something like this:

Log Collections Settings

  1. In Logs Data Provider, select the custom logs provider you added, described in 1 – Custom Verification Connection Setup.
  2. Click Add Collection Details to add a Logs Collection section.
  • In Log Collection URL, enter the API query that will return a JSON response. When you enter a query, the Response Mapping settings appear. In this section, you will map the keys in the JSON response to Harness fields, to identify where data—such as the hostname and timestamp—are located in the JSON response.
  • In Response Type, select JSON.
  • In Metrics Method, select GET or POST. If you select POST, you also enable a Log Collection Body field where you can enter any HTTP body to send as part of the query.
  • In Log Message JSON Path – Use Guide from an example to query the log provider and return the JSON response.

The URL is a combination of the Verification Cloud Provider Base URL and the Log Collection URL you entered.

Click SEND. In the JSON response, click the key that includes the log message path.

The log message path key is added to Log Message JSON Path:

  • Hostname JSON Path -–Use Guide from an example to query the log provider and return the JSON response. In the JSON response, click the key that includes the hostname path.

  • Regex to transform Host Name – If the JSON value returned requires transformation in order to be used, enter the regex expression here. For example: If the value in the host name JSON path of the response is pod_name:harness-test.pod.name and the actual pod name is simply harness-test.pod.name, you can write a regular expression to remove the pod_name from the response value.
  • Timestamp JSON path – Use Guide from an example to query the log provider and return the JSON response. In the JSON response, click the key that includes the timestamp.

  • Timestamp Format – Enter the format of the timestamp included in the query request (not response). The format follows the Java SimpleDateFormat. For example, a timestamp syntax might be yyyy-MM-dd'T'HH:mm:ss.SSSX. If you leave this field empty, Harness will use the default range of 1 hour previous (now - 1h) to now.

Click ADD. The Log Collection is added.

Custom Thresholds

In the Configure Custom Metrics dialog, you can access the Custom Thresholds section once you have configured at least one Metrics Collection. You can use Custom Thresholds to define two types of rules that override normal verification behavior:

  • Ignore Hints that instruct Harness to skip certain metrics/value combinations from verification analysis.
  • Fast-Fail Hints that cause a Workflow to enter a failed state.
Custom Thresholds are currently behind a feature flag. Contact Harness Support to enable this feature.

To define Custom Thresholds:

  1. Click the pencil icon shown below.
  2. In the resulting dialog, select either the Ignore Hints or the Fast-Fail Hints tab.
  3. Click Add Threshold to define a rule.
  4. Use the drop-downs to select a desired Transaction Name and Metric Name from your defined Metrics Collections.
  5. For Fast-Fail Hints, select an Action to take: Fail immediately, Fail after multiple occurrences, or Fail after multiple occurrences in a row. Two of these selections will expose a field where you must specify the threshold Occurrence Count.
  6. Select the Criteria for this rule, and enter a corresponding Value.

    For various Criteria selections, the Value field's label will change to Less than for Ignore Hints, and to Greater than and/or Less than selectors for Fast-Fail Hints (as shown below).
    Here are the Criteria and Value options available for the metric you've selected.

    Criteria

    Value

    Absolute Value

    Enter a literal value of the selected metric. In Ignore Hints, observed values Less than this threshold will be skipped from verification analysis. In Fast-Fail Hints, use the Range Selector drop-down to select whether observed values Less than or Greater than your threshold Value will move the Workflow to a failed state.

    Percentage Deviation

    Enter a threshold percentage at which to either skip the metric from analysis (Ignore Hints), or fail the Workflow (Fast-Fail Hints). Units here are percentages, so entering 3 will set the threshold at a 3% anomaly from the norm.

    Deviation

    This also sets a threshold deviation from the norm. But here, the units are not percentages, but literal values of the selected metric.

  7. Click Add Threshold to define additional rules.
  8. Click Submit to save and apply your rules.
In deployment, where a Fast-Fail Hint moves a Workflow to a failed state, the Workflow's Details panel for that Verification step will indicate the corresponding threshold.

Common Settings

The following settings are common to both metrics and log providers.

Field

Description

Expression for Host/Container

The expression entered here should resolve to a host/container name in your deployment environment. By default, the expression is ${host.hostName}.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Harness waits 2-3 minutes before beginning the analysis to avoid initial deployment noise. This is a standard with monitoring tools.

Baseline for Risk Analysis

See CV Strategies, Tuning, and Best Practices.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

Include instances from previous phases

If you are using this verification step in a multi-phase deployment, select this checkbox to include instances used in previous phases when collecting data. Do not apply this setting to the first phase in a multi-phase deployment.

Wait interval before execution

Set how long the deployment process should wait before executing the verification step.

Notes

  • Depending on the custom metric provider you select, you might need to provide different information to the Metric Collections section. For example, you might need to provide a hostname for the Guide from an example popover to use to retrieve data. The hostname will be the host/container/pod/node name where the artifact is deployed. In you look in the JSON for the deployment environment, the hostname is typically the name label under the host label.

  • The Compare With Previous Run option is used for Canary deployments where the second phase is compared to the first phase, and the third phase is compared to the second phase, and so on. Do not use this setting in a single phase workflow or in the first phase of a multi-phase workflow.

Next Step


How did we do?