Prometheus Verification

Updated 1 month ago by Michael Cretzman

Prometheus uses a multi-dimensional data model with time series data and key/value pairs, along with a flexible query language to leverage this dimensionality. Prometheus records any numeric time series, such as machine-centric monitoring and the monitoring of highly dynamic service-oriented architectures. For microservices, Prometheus support for multi-dimensional data collection and querying is very useful.

Prometheus integrates with Harness to verify the performance of microservices instantly in every environment. When you deploy a new microservice, Harness automatically connects to Prometheus and starts analyzing the multi-dimensional data model to understand what exceptions and errors are new or might cause problems for your microservice performance and quality.

Intended Audience

  • DevOps

Before You Begin

Connect to Prometheus

To add Prometheus as a verification provider, do the following:

  1. Click Setup.
  2. Click Connectors.
  3. Click Verification Providers.
  4. Click Add Verification Provider, and select Prometheus. The Add Prometheus Verification Provider dialog appears.

The Add Prometheus Verification Provider dialog has the following fields.

Field

Description

URL

Enter the URL of the server.

You cannot use a Grafana URL.

Display Name

Enter a display name for the provider. If you are going to use multiple providers of the same type, ensure you give each provider a different name.

Usage Scope

If you want to restrict the use of a provider to specific applications and environments, do the following:

In Usage Scope, click the drop-down under Applications, and click the name of the application.

In Environments, click the name of the environment.

Verify with Prometheus

The following procedure adds a Prometheus verification step to a workflow.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

To verify your deployment with Prometheus, do the following:

  1. Ensure that you have added Prometheus as a verification provider, as described above.
  2. In your workflow, under Verify Service, click Add Verification, and then click Prometheus. The Prometheus dialog appears.

The Prometheus workflow dialog has the following fields.

Field

Description

Prometheus Server

Select the server you added when setting up the Prometheus verification provider.

Metric to Monitor

Every time series is uniquely identified by its metric name and a set of key-value pairs, also known as labels. For more information, see Data Model from Prometheus.
A metric requires the following parameters:

  • Transaction Name: The service or request context which the metric relates to. For example, Login.
  • Metric Name: The name of the metric defined in Prometheus.
  • Metric Type: The type of metric (Response Time, Error, Throughput, or Value).
  • URL: The API query required to retrieve the metric value. This query must include placeholders for start time, end time, and hostname.

Expression for Host/Container Name

The expression entered here should resolve to a host/container name in your deployment environment. By default, the expression is ${host.hostName}. If you begin typing the expression into the field, the field provides expression assistance.

The following settings are common to all verification provider dialogs in workflows.

Field

Description

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline for Risk Analysis

Select Previous Analysis to have this verification use the previous analysis for a baseline comparison. If your workflow is a Canary workflow type, you can select Canary Analysis to have this verification compare old versions of nodes to new versions of nodes in realtime.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

Include instances from previous phases

If you are using this verification step in a multi-phase deployment, select this checkbox to include instances used in previous phases when collecting data. Do not apply this setting to the first phase in a multi-phase deployment.

Wait interval before execution

Set how long the deployment process should wait before executing the verification step.

Verification Results

When Harness deploys a new application or service to the target environment defined in the workflow, it will immediately connect to the Prometheus Server and build a model of what it is observing.

Next, Harness compares this model with previous deployment models to identify anomalies or regressions. If necessary, Harness rolls back to the previous working version automatically. For more information, see Rollback Steps.

Here is an example of a typical, successful deployment verified using Jenkins and Prometheus.

Under Prometheus, you can see that all Prometheus metrics have been validated by the Harness machine learning algorithms. Green indicates that there are no anomalies or regressions identified and the deployment is operating within its normal range.

To see an overview of the verification UI elements, see Continuous Verification Tools.


How did we do?