Splunk Verification

Updated 3 days ago by Michael Cretzman

The following sections describe how Harness integrates ​Splunk into Harness Continuous Verification to monitor your live, production services and verify your deployments.

Splunk and Harness

Splunk Enterprise enables you to search, analyze, and visualize data gathered from your microservices, websites, and apps. After you define the data source, Splunk Enterprise indexes the data stream and parses it into a series of individual events that you can view and search. Splunk provides a REST API with over 200 endpoints. Developers can programmatically index, search, and visualize data in Splunk from any app.

Once you have integrated Splunk with your microservice or app, you can add a Splunk verification step to your Harness workflows and Harness will use Splunk to verify the performance and quality of your deployments and apply Harness machine-learning verification analysis to Splunk data.

Verification with Splunk Enterprise

Harness Analysis

Setup Preview

You set up Splunk and Harness in the following way:

  1. Splunk - Monitor your application using Splunk. In this article, we assume that you are using Splunk to monitor your application already.
  2. ​Verification Provider Setup - In Harness, you connect Harness to your Splunk account, adding Splunk as a Harness Verification Provider.
  3. Harness Application - Create a Harness Application with a Service and an Environment. We do not cover Application set up in this article. See Application Components.
  4. ​24/7 Service Guard Setup- In the Environment, set up 24/7 Service Guard to monitor your live, production application.
  5. Verify Deployments:
    1. Add a Workflow to your Harness Application and deploy your microservice or application to the service infrastructure in your Environment.
    2. After you have run a successful deployment, you then add verification steps to the Workflow using your Verification Provider.
    3. Harness uses unsupervised machine-learning and Splunk analytics to analyze your future deployments, discovering events that might be causing your deployments to fail. Then you can use this information to set rollback criteria and improve your deployments.

Before You Begin

Permissions for API Connection

Splunk APIs require that you authenticate with a non-SAML account. To access your Splunk Cloud deployment using the Splunk REST API and SDKs, submit a support case requesting access on the Support Portal. For managed deployments, Splunk Support opens port 8089 for REST access. You can specify a range of IP addresses to control who can access the REST API. For self-service deployments, Splunk Support defines a dedicated user and sends you credentials that enable that user to access the REST API. For information on Splunk self-service accounts, see Using the REST API with Splunk Cloud.

Ensure the Splunk user account used to authenticate Harness with Splunk has the following rest-related capabilities:

  • rest_apps_view
  • rest_properties_get
  • rest_properties_set
  • rest_apps_management

Verification Provider Setup

The first step in using Splunk with Harness is to set up an Splunk Verification Provider in Harness.

A Harness Verification Provider is a connection to monitoring tools such as Splunk. Once Harness is connected, you can use Harness 24/7 Service Guard and Deployment Verification with your Splunk data and analysis.

To add Datadog as a verification provider, do the following:

  1. Click Setup.
  2. Click Connectors.
  3. Click Verification Providers.
  4. Click Add Verification Provider, and select Splunk. The Add Splunk Verification Provider dialog for your provider appears.

The Add Splunk Verification Provider dialog has the following fields.




Enter the URL for accessing the REST API on the Splunk server. Include the port number in the format https://<deployment-name>.cloud.splunk.com:8089: The default port number is 8089.The port number is required for hosted Splunk, also. For example: https://mycompany.splunkcloud.com:8089.For more information, see Using the REST API with Splunk Cloud from Splunk.

Username and Password

Enter the account credentials to authenticate with the server. A user role that is not authenticated with SAML is required. You do not need an admin role.

Display Name

Enter a display name for the provider. If you are going to use multiple providers of the same type, ensure you give each provider a different name.

Usage Scope

If you want to restrict the use of a provider to specific applications and environments, do the following:

  1. In Usage Scope, click the drop-down under Applications, and click the name of the application.
  2. In Environments, click the name of the environment.

Verify with Splunk

The following procedure describes how to add Splunk as a verification step in a Harness workflow. For more information about workflows, see Add a Workflow.

Once you run a deployment and Splunk preforms verification, Harness' machine-learning verification analysis will assess the risk level of the deployment.

In order to obtain the names of the host(s), pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment.

To verify your deployment with Splunk, do the following:

  1. Ensure that you have added Splunk as a verification provider, as described above.
  2. In your workflow, under Verify Service, click Add Verification, and then click Splunkv2. The Splunk V2 dialog appears.

The Splunk dialog has the following fields.



Splunk Server

Select the Harness Verification Provider you configured using your Splunk account.

Search Keywords

Enter a search term or query. To search for all exceptions, use asterisks (*) around exception, for example, *exception*. For more information, see Retrieve events from the index from Splunk.

When you enter a search such as *exception*, at runtime Harness will generate a query containing your search and information Harness needs to perform verification, such as the information following *exception* below:

search *exception* host = ip-172-31-81-88 | bin _time span=1m | 
cluster t=0.9999 showcount=t labelonly=t|
table _time, _raw,cluster_label, host |
stats latest(_raw) as _raw count as cluster_count by _time,cluster_label,host

If you want more flexibility in your search, or to repurpose Splunk searches you already have, you can click Advanced Query and enter whatever you like in Search Keywords. For example, you could replace *exception* with an existing Splunk search.

Note that you will specify host field name and host/pod/container name in other settings so you do not need to include them in the search query.

Field name for Host/Container

Typically, you will enter host. You can enter host into the Splunk Search field to see the host for your Harness deployment:

Expression for Host/Container name

See Guide From Example.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline for Risk Analysis

Select Previous Analysis to have this verification use the previous analysis for a baseline comparison. If your workflow is a Canary workflow type, you can select Canary Analysis to have this verification compare old versions of nodes to new versions of nodes in real-time.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Guide From Example

In the Splunk verification step dialog, you can see the Guide From Example option next to the Expression for Host/Container name field. This option lets you select the host(s), pod(s), or container(s) for Harness to use when performing verification.

You select the host, pod, or container in Guide From Example, and an expression is added to the Expression for Host/Container name field. The default expression is ${host.hostName}. Typically, you can simply use ${host.hostName}.

In order to obtain the names of the host(s) pod(s), or container(s) where your service is deployed, the verification provider should be added to your workflow after you have run at least one successful deployment. Then the Guide From Example feature can display the host or container name(s) for you to select.

To ensure that you pick the right name when using Guide From Example, you can use a host name in Splunk as a guide.

To use Guide From Example for a host or container name expression, do the following:

  1. In Splunk, click App: Search & Reporting, and then click Search & Reporting.
  2. In Search, enter host to see a list of the available hosts being tracked.
  3. Click the name of your host to add it to the search, select a date range, and click the search icon. The event log entries for the host appear.
    The name of the host can be seen in the event message, next to host =. The expression that you provide in the Expression for Host/Container Name field in the Harness Splunk dialog should evaluate to the name here.
  4. In your Harness workflow Splunk dialog, click Guide From Example. The Expression for Host Name popover appears.
    The dialog shows the service, environment, and service infrastructure used for this workflow.
  5. In Host, click the name of the host to use when testing verification. Match the hostname from the Splunk Search to the hostname in the Expression for Host Name popover:
  6. Click SUBMIT. The YAML for the host appears. Look for the host section.
    You want to use a hostName label in the host section. Do not use a hostName label outside of that section.
  7. Click the hostName label. The variable name is added to the Expression for Host/Container name field.
  8. At the bottom of the Splunk dialog, click TEST. A new Expression for Host Name popover appears.
  9. In Host, select the same host you selected last time, and then click RUN. Verification information for the host is found. In there is no verification data for the selected node, the test will display connection information only.
  10. Click back in the Splunk dialog and click SUBMIT. The Splunk verification step is added to your workflow.

Verification Results

Once you have deployed your workflow (or pipeline) using the Splunk verification step, you can automatically verify app performance across your deployment. For more information, see Add a Workflow and Add a Pipeline.

Workflow Verification

After you add the Splunk verification step to your workflow, the next time you deploy the workflow you will see the Splunk verification step running:

To see the results of Harness machine-learning evaluation of your Splunk verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the Splunk step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

To learn about the verification analysis features, see the following sections.


Deployment infoSee the verification analysis for each deployment, with information on its service, environment, pipeline, and workflows.

Verification phases and providersSee the vertfication phases for each vertfication provider. Click each provider for logs and analysis.

Verification timelineSee when each deployment and verification was performed.

Transaction Analysis

Execution detailsSee the details of verification execution. Total is the total time the verification step took, and Analysis duration is how long the analysis took.

Risk level analysisGet an overall risk level and view the cluster chart to see events.

Transaction-level summarySee a summary of each transaction with the query string, error values comparison, and a risk analysis summary.

Execution Analysis

Event typeFilter cluster chart events by Unknown Event, Unexpected Frequency, Anticipated Event, Baseline Event, and Ignore Event.

Cluster chartView the chart to see how the selected event contrast. Click each event to see its log details.

Event Management

Event-level analysisSee the threat level for each event captured.

Tune event captureRemove events from analysis at the service, workflow, execution, or overall level.

Event distributionClick the chart icon to see an event distribution including the measured data, baseline data, and event frequency.

Next Steps

How did we do?