Bugsnag Verification

Updated 3 days ago by Michael Cretzman

The following sections describe how Harness integrates Bugsnag into Harness Continuous Verification to monitor your live, production services and verify your deployments:

Bugsnag and Harness

Bugsnag provides error reporting libraries for every major software platform, detecting and reporting errors in your apps, and capturing diagnostic data for each error. Bugsnag captures your app’s exceptions or events and groups them into errors according to their root causes.

Harness Continuous Verification integrates with Bugsnag to verify your deployments and live production applications using the following Harness features:

  • 24/7 Service Guard - Monitors your live, production applications.
  • Deployment Verification - Monitors your application deployments, and performs automatic rollback according to your criteria.

This document describes how to set up these Harness Continuous Verification features and monitor your deployments and production applications using its unsupervised machine-learning functionality.

Bugsnag provides unique browser-focused reporting. See Browser-Based Benefits below.

Reporting with Bugsnag

Harness Analysis

Setup Preview

You set up Bugsnag and Harness in the following way:

  1. Bugsnag - Monitor your application using Bugsnag. In this article, we assume that you are using Bugsnag to monitor your application already.
  2. ​Verification Provider Setup - In Harness, you connect Harness to your Bugsnag account, adding Bugsnag as a Harness Verification Provider.
  3. Harness Application - Create a Harness Application with a Service and an Environment. We do not cover Application set up in this article. See Application Checklist.
  4. ​24/7 Service Guard Setup- In the Environment, set up 24/7 Service Guard to monitor your live, production application.
  5. Verify Deployments:
    1. Add a Workflow to your Harness Application and deploy your microservice or application to the service infrastructure in your Environment.
    2. After you have run a successful deployment, you then add verification steps to the Workflow using your Verification Provider.
    3. Harness uses unsupervised machine-learning and Bugsnag analytics to analyze your future deployments, discovering events that might be causing your deployments to fail. Then you can use this information to set rollback criteria and improve your deployments.

Browser-Based Benefits

Bugsnag is particularly useful for browser-based apps, as it collects browser information as part of its exception and error capture. This can helpful in determining if a new version of a browser is causing problems for users. Here is an example from Bugsnag:

Once you have deployed your app via Harness, you can add host/node-focused verification to your Harness workflow using another Verification Provider, and use Bugsnag to focus on browser-based issues. Here is an example of a Harness verification where other verification tools have been used to verify host/node issues, and Bugsnag is added as the last verification step to capture browser-based issues:

When you set up Bugsnag as a verification step in a Harness workflow, you can indicate if your app is browser-based. When Harness arrives at the Bugsnag verification step, Harness will ignore deployment host or node information and focus on browser-based data. This browser focus enables you to capture browser issues on their own after you have already ensured that the deployment host/node environment is running correctly.

Harness can now use this browser data with the machine-learning in its Continuous Verification and determine what events are causing errors or have to the potential to cause errors in the future.

For information about advanced browser event capturing in Bugsnag, see React integration guide from Bugsnag.

Verification Provider Setup

The first step in using Bugsnag with Harness is to set up an Bugsnag Verification Provider in Harness.

A Harness Verification Provider is a connection to monitoring tools such as Bugsnag. Once Harness is connected, you can use Harness 24/7 Service Guard and Deployment Verification with your Bugsnag data and analysis.

To add Bugsnag as a verification provider, do the following:

  1. Click Setup.
  2. Click Connectors, and then click Verification Providers.
  3. Click Add Verification Provider, and select Bugsnag. The Add Bugsnag Verification Provider dialog for your provider appears.

The Add Bugsnag Verification Provider dialog has the following fields.

Field

Description

Bugsnag URL

Enter https://api.bugsnag.com/. This is the URL for the Bugsnag API.

Auth Token

Enter the Data Access API personal auth token for your Bugsnag account. Here is how you get the token in Bugsnag:

  1. In Bugsnag, click your account icon in the top-right of any page, and then click Settings.
  2. In the My account page, next to Data Access API, click Personal auth tokens.
  3. In Personal auth tokens, click GENERATE NEW TOKEN.
    The Generate new auth token dialog appears.
  4. In Token description, enter a name, such as Harness, and click GENERATE. The auth token is generated.
  5. Click Copy to clipboard.
  6. In Harness, in the Add Bugsnag Verification Provider dialog, paste the token in the Auth Token field.

Display Name

The name for this Bugsnag verification provider connection in Harness. If you will have multiple Bugsnag connections, enter a unique name. You will use this name to select this connection when integrating Bugsnag with the Verify Steps of your workflows, described below.

Usage Scope

If you want to restrict the use of a provider to specific applications and environments, do the following:

In Usage Scope, click the drop-down under Applications, and click the name of the application.

In Environments, click the name of the environment.

24/7 Service Guard Setup

Harness 24/7 Service Guard monitors your live applications, catching problems that surface minutes or hours following deployment. For more information, see 24/7 Service Guard.

You can add your Bugsnag monitoring to Harness 24/7 Service Guard in your Harness Application Environment. For a setup overview, see Setup Preview.

This section assumes you have a Harness Application set up and containing a Service and Environment. For steps on setting up a Harness Application, see Application Checklist.

To set up 24/7 Service Guard for Bugsnag, do the following:

  1. Ensure that you have added Bugsnag as a Harness Verification Provider, as described in Verification Provider Setup.
  2. In your Harness Application, ensure that you have added a Service, as described in Services. For 24/7 Service Guard, you do not need to add an Artifact Source to the Service, or configure its settings. You simply need to create a Service and name it. It will represent your application for 24/7 Service Guard.
  3. In your Harness Application, click Environments.
  4. In Environments, ensure that you have added an Environment for the Service you added. For steps on adding an Environment, see Environments.
  5. Click the Environment for your Service. Typically, the Environment Type is Production.
  6. In the Environment page, locate 24/7 Service Guard.
  7. In 24/7 Service Guard, click Add Service Verification, and then click Bugsnag.
    The Bugsnag dialog appears.
  8. Fill out the dialog. The Bugsnag dialog has the following fields.
For 24/7 Service Guard, the queries you define to collect logs are specific to the application or service you want monitored. Verification is application/service level. This is unlike Workflows, where verification is performed at the host/node/pod level.

Field

Description

Display Name

The name that will identify this service on the Continuous Verification dashboard. Use a name that indicates the environment and monitoring tool, such as Bugsnag.

Service

The Harness Service to monitor with 24/7 Service Guard.

Bugsnag Server

Select the Bugsnag verification provider you added to Harness, as described above. Harness will immediately use the connection to obtain organization and project information from Bugsnag.

Organization ID

Select the Organization ID for your Bugsnag account. In Bugsnag, this is the Organization name in the account's Organization page:

Project ID

Select the Project ID for the Bugsnag project you want to use. In Bugsnag, this is the Project Name in the Projects page:

Release Stage

Enter the Bugsnag release stage, if necessary.

Search Keywords

The keywords to search, such as *exception*.

Browser Application

Click the checkbox to have Harness ignore host/node events and focus on the browser events Bugsnag captures.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline

Select the baseline time unit for monitoring. For example, if you select For 4 hours, Harness will collect the logs for the last 4 hours as the baseline for comparisons with future logs. If you select Custom Range you can enter a Start Time and End Time.

When you are finished, the dialog will look something like this:

Click TEST. Harness verifies the settings you entered.

Click SUBMIT. The ELK 24/7 Service Guard is configured.

To see the running 24/7 Service Guard analysis, click Continuous Verification. The 24/7 Service Guard dashboard displays the production verification results.

For information on using the dashboard, see Using 24/7 Service Guard.

Verify Deployments

The following procedure describes how to add Bugsnag as a verification step in a Harness workflow. For more information about workflows, see Add a Workflow.

Once you run a deployment and Bugsnag preforms verification, Harness' machine-learning verification analysis will assess the risk level of the deployment.

In order to obtain the useful comparison data, the verification provider should be added to your workflow after you have run at least one successful deployment.

Deployment Verification Setup

To verify your deployment with Bugsnag, do the following:

  1. Ensure that you have added Bugsnag as a verification provider, as described above.
  2. In your workflow, under Verify Service, click Add Verification, and then click Bugsnag. The Bugsnag dialog appears.

The Bugsnag dialog has the following fields.

Field

Description

Bugsnag Server

Select the Bugsnag verification provider you added to Harness, as described above. Harness will immediately use the connection to obtain organization and project information from Bugsnag.

Organization ID

Select the Organization ID for your Bugsnag account. In Bugsnag, this is the Organization name in the account's Organization page:

Project ID

Select the Project ID for the Bugsnag project you want to use. In Bugsnag, this is the Project Name in the Projects page:

Release Stage

Enter the Bugsnag release stage, if necessary.

Browser Application

Click the checkbox to have Harness ignore host/node deployment events and focus on the browser events Bugsnag captures.

Expression for Host/Container name

If you do not enable the Browser Application checkbox, Harness will use the host/node/container event data Bugsnag captures.

Add a variable that evaluates to the hostname value in the host field of event messages. For example, in a Bugsnag message in a Harness deployment verification, if you look at a event message, you will see a hosts field:

Next, look in the JSON for the host/container/pod in the deployment environment and identify the label containing the same hostname. The path to that label is what the expression should be in Expression for Host/Container name. The default variable is ${host.hostName}. In most cases, this expression will work.

Analysis Time duration

Set the duration for the verification step. If a verification step exceeds the value, the workflow Failure Strategy is triggered. For example, if the Failure Strategy is Ignore, then the verification state is marked Failed but the workflow execution continues.

Baseline for Risk Analysis

Select Previous Analysis to have this verification use the previous analysis for a baseline comparison. If your workflow is a Canary workflow type, you can select Canary Analysis to have this verification compare old versions of nodes to new versions of nodes in real-time.

Execute with previous steps

Check this checkbox to run this verification step in parallel with the previous steps in Verify Service.

Failure Criteria

Specify the sensitivity of the failure criteria. When the criteria is met, the workflow Failure Strategy is triggered.

Include instances from previous phases

If you are using this verification step in a multi-phase deployment, select this checkbox to include instances used in previous phases when collecting data. Do not apply this setting to the first phase in a multi-phase deployment.

When you are finished, click SUBMIT. The Bugsnag verification step is added to your workflow.

Verification Results

Once you have deployed your workflow (or pipeline) using the Bugsnag verification step, you can automatically verify cloud application and infrastructure performance across your deployment. For more information, see Add a Workflow and Add a Pipeline.

Workflow Verification

To see the results of Harness machine-learning evaluation of your Bugsnag verification, in your workflow or pipeline deployment you can expand the Verify Service step and then click the Bugsnag step.

Continuous Verification

You can also see the evaluation in the Continuous Verification dashboard. The workflow verification view is for the DevOps user who developed the workflow. The Continuous Verification dashboard is where all future deployments are displayed for developers and others interested in deployment analysis.

To learn about the verification analysis features, see the following sections.

Deployments

Deployment info
See the verification analysis for each deployment, with information on its service, environment, pipeline, and workflows.

Verification phases and providers
See the vertfication phases for each vertfication provider. Click each provider for logs and analysis.

Verification timeline
See when each deployment and verification was performed.

Transaction Analysis

Execution details
See the details of verification execution. Total is the total time the verification step took, and Analysis duration is how long the analysis took.

Risk level analysis
Get an overall risk level and view the cluster chart to see events.

Transaction-level summary
See a summary of each transaction with the query string, error values comparison, and a risk analysis summary.

Execution Analysis

Event type
Filter cluster chart events by Unknown Event, Unexpected Frequency, Anticipated Event, Baseline Event, and Ignore Event.

Cluster chart
View the chart to see how the selected event contrast. Click each event to see its log details.

Event Management

Event-level analysis
See the threat level for each event captured.

Tune event capture
Remove events from analysis at the service, workflow, execution, or overall level.

Event distribution
Click the chart icon to see an event distribution including the measured data, baseline data, and event frequency.

Next Steps


How did we do?