3 - Verify Deployments with Elasticsearch

Updated 3 weeks ago by Michael Cretzman

Harness can analyze Elasticsearch data and analysis to verify, rollback, and improve deployments. To apply this analysis to your deployments, you set up Elasticsearch as a verification step in a Harness Workflow.

This section covers how to set up Elasticsearch in a Harness Workflow, and provides a summary of Harness verification results.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

In this topic:

Deployment Verification Setup

To add an ELK verification step to your Workflow, do the following:

  1. Ensure that you have added ELK Elasticsearch as a Verification Provider, as described in Verification Provider Setup.
  2. In your Workflow, under Verify Service, click Add Verification, and then click ELK. The ELK dialog appears.

To configure the ELK dialog fields, do the following:

  1. In Elastic Search Server, select the server you added when you set up the ELK verification provider, as described above.
  2. In Search Keywords, enter search keywords for your query, such as error or exception. The keywords are searched against the logs identified in the Message field of the dialog (see below).

    For an advanced query, enter an Elasticsearch JSON query. You can use JSON to create complex queries beyond keywords. The following example looks for the substring error in the field log:

    {"regexp":{"log": {"value":"error"}}}
  3. In Query Type, select TERM to finds documents that contain the exact term specified in the inverted index. MATCH queries accept text, numerics, and dates, analyze them, and construct a query. If you want the query analyzed, then use MATCH.
  4. In Index, enter the the index to search. This field is automatically populated from the index templates, if available.
    If there are no index templates, or if you do not have administrator privileges with ELK, enter the index manually.
    1. To locate indices, in Kibana, click Management.
    2. Click Index Patterns. The Index Patterns page appears.
    3. Copy the name of one of the Index patterns.
    4. In Harness, in the ELK dialog, paste the name of the Index pattern into Indices.
  5. In Host Name Field, enter the field name used in the ELK logs that refers to the host/pod/container ELK is monitoring.

    To find the hostname in Kibana and enter it in Harness, do the following:
    1. In Kibana, click Discover.
    2. In the search field, search for erroror exception.
    3. In the results, locate the host name of the host/container/pod where ELK is monitoring. For example, when using Kubernetes, the pod name field kubernetes.pod_name is used.
    4. In Harness, in the ELK dialog, next to Host Name Field, click Guide From Example. The Host Name Field popover appears.
    5. In the JSON response, click on the name of the label that maps to the host/container/pod in your log search results. Using our Kubernetes example, under pod, you would click the first name label.

      The Host Name Field is filled with the JSON label for the hostname.
  6. In Message Field, enter the field by which the messages are usually indexed. Typically, a log field.

    To find the field in Kibana and enter it in Harness, do the following:
    1. In Kibana, click Discover.
    2. In the search field, search for error or exception.
    3. In the results, locate a log for the host/container/pod ELK is monitoring. For example, in the following Kubernetes results in Kibana, the messages are indexed under the log field.
    4. In Harness, in the ELK dialog, next to Message Field, click Guide From Example. The Message Field popover appears.
    5. In the JSON response, click on the name of the label that maps to the log in your Kibana results. Using our Kubernetes example, you would click the log label.

      The label is added to the Message Field.
  7. In Expression for Host/Container name, add an expression that evaluates to the host name value for the field you entered in the Host Name Field above. The default expression is ${host.hostName}.
    In order to obtain the names of the host where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.
    To ensure that you pick the right field when using Guide From Example, you can use a host name from the ELK log messages as a guide.

    To use Guide From Example for a host name expression, do the following:
    1. In Kibana, click Discover.
    2. In the search field, search for error or exception.
    3. In the results, locate the name of the host/container/pod ELK is monitoring. For example, when using Kubernetes, the pod name field kubernetes.pod_name displays the value you need.

      The expression that you provide in Expression for Host/Container Name should evaluate to the name here, although the suffixes can differ.
    4. In Harness, in your workflow ELK dialog, click Guide From Example. The Expression for Host Name popover appears.

      The dialog shows the service, environment, and service infrastructure used for this workflow.
    5. In Host, click the name of the host to use when testing verification. The hostname will be similar to the hostname you used for the Host Name Field, as described earlier in this procedure. The suffixes can be different.
    6. Click SUBMIT. The JSON for the host appears. Look for the host section.

      You want to use a name label in the host section. Do not use a host name label outside of that section.
    7. To identify which label to use to build the expression, compare the host/pod/container name in the JSON with the hostname you use when configuring Host Name Field.
    8. In the Expression for Host Name popover, click the name label to select the expression. Click back in the main dialog to close the Guide From Example. The expression is added to the Expression for Host/Container name field.

      For example, if you clicked the name label, the expression ${host.name} is added to the Expression for Host/Container name field.
  8. In Timestamp format, enter the format for the timestamp field in the Elasticsearch record. Use Kibana to determine the format.

    In Kibana, use the Filter feature in Discover to construct your timestamp range:
    Format Examples:

    Timestamp: 2018-08-24T21:40:20.123Z. Format: yyyy-MM-dd'T'HH:mm:ss.SSSX

    Timestamp: 2018-08-30T21:57:23+00:00. Format: yyyy-MM-dd'T'HH:mm:ss.SSSXXX

    For more information, see Data Math from Elastic.
  9. At the bottom of the New Relic dialog, click TEST.

    A new Expression for Host Name popover appears.

    In Host, select the same host you selected last time, and then click RUN. Verification for the host is found.

If you receive an error, it is likely because you selected the wrong label in Expression for Host/Container name or Host Name Field.

  1. Next, click Analysis Details. The Analysis Details appear.

The following settings are common to all verification provider dialogs in workflows.

Field

Description

Analysis Period

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Harness waits 2-3 minutes before beginning the analysis to avoid initial deployment noise. This is a standard with monitoring tools.

Baseline for Risk Analysis

See CV Strategies, Tuning, and Best Practices.

For Canary Analysis and Previous Analysis, analysis happens at the host/node/pod level. For Predictive Analysis, data collection happens at the host/node/pod level but analysis happens at the application or service level. Consequently, for data collection, provide a query that targets the logs for the host using fields such as SOURCE_HOST in Field name for Host/Container.

Algorithm Sensitivity

Select the sensitivity that will result in the most useful results for your analysis.

See CV Strategies, Tuning, and Best Practices.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Include instances from previous phases

If you are using this verification step in a multi-phase deployment, select this checkbox to include instances used in previous phases when collecting data. Do not apply this setting to the first phase in a multi-phase deployment.

When you are finished, click SUBMIT. The ELK verification step is added to your workflow.

Templatize ELK Verification

Once you have created an ELK verification step, you can templatize certain settings. This enables you to use the ELK verification step in the Workflow (and multiple Pipelines) without having to provide settings until runtime.

You templatize settings by click the [T] icon next to the setting.

The settings are replaced by Workflow variables:

You will now see them in the Workflow Variables section of the Workflow:

When you deploy the Workflow, Start New Deployment prompts you to enter values for templatize settings:

You can select the necessary settings and deploy the Workflow.

You can also pass variables into a Workflow from a Trigger that can be used for templatized values. For more information, see Passing Variables into Workflows and Pipelines from Triggers.

Verification Results

Once you have deployed your workflow (or pipeline) using the New Relic verification step, you can automatically verify cloud application and infrastructure performance across your deployment.

Workflow Verification

To see the results of Harness machine-learning evaluation of your ELK verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the ELK step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

To learn about the verification analysis features, see the following sections.

Deployments

Deployment infoSee the verification analysis for each deployment, with information on its service, environment, pipeline, and workflows.

Verification phases and providersSee the vertfication phases for each vertfication provider. Click each provider for logs and analysis.

Verification timelineSee when each deployment and verification was performed.

Transaction Analysis

Execution detailsSee the details of verification execution. Total is the total time the verification step took, and Analysis duration is how long the analysis took.

Risk level analysisGet an overall risk level and view the cluster chart to see events.

Transaction-level summarySee a summary of each transaction with the query string, error values comparison, and a risk analysis summary.

Execution Analysis

Event typeFilter cluster chart events by Unknown Event, Unexpected Frequency, Anticipated Event, Baseline Event, and Ignore Event.

Cluster chartView the chart to see how the selected event contrast. Click each event to see its log details.

Event Management

Event-level analysisSee the threat level for each event captured.

Tune event captureRemove events from analysis at the service, workflow, execution, or overall level.

Event distributionClick the chart icon to see an event distribution including the measured data, baseline data, and event frequency.

Next Step


How did we do?