Diagnose Delegate Issues

Updated 2 weeks ago by Michael Cretzman

To diagnose Harness Delegate issues, you can use the general troubleshooting steps in this topic.

For information on general troubleshooting, see Troubleshooting.

To download or export deployment logs, which contain deployment events and errors, but not many Delegate events and errors, see Export Deployment Logs.

In this topic:

Before You Begin

Step 1: Look at Common Causes

The following steps will help you diagnose many issues.

  1. Troubleshoot the Delegate validation URL. When a task is ready to be assigned, the Harness Manager first validates its lists of Delegates to see which Delegate should be assigned the task. It validates the Delegate using the URL in the task, such as a API call or SSH command. See How Does Harness Manager Pick Delegates?.
    1. Locate the URL by looking at the Delegate log.
    2. See if that URL is reachable from the Delegate host or from another host. Log into the host running the Delegate and see if the URL can be reached via cURL or ping.
  2. Ensure that the Delegate is not blacklisted. If a Delegate fails to perform a task that Delegate is blacklisted for that task and will not be tried again. TTL is 5 minutes. This is true if there is only one Delegate and even if the Delegate is Tagged for that task, such as with a Shell Script command in a Workflow.
  3. Use all Delegate Tags. If you are using Delegate Tags for a task (via Shell Script command) or Cloud Provider credentials, ensure that the command or provider has all of the Tags used by the Delegate. If the command or provider has less than all of the Tags used by the Delegate, the tagging will not work.
  4. Review Delegate Scoping. Ensure the Delegate is not scoped out of performing the task. See Delegate Scope and Best Practices and Notes.
  5. Delegate Restarting. Check to see if the Delegate is restarting (for example, because the socket connection is failing) and offline at the time of the task.

Step 2: Obtain Execution ID from the Deployment URL

You can diagnose a lot of Delegate issues using the Execution ID, Task ID and Delegate information from the task assignment and operation.

A deployment has an Execution ID. Every Execution ID includes one or more tasks, identified by Task IDs. Using the Task ID, you can determine which Delegate was assigned the task and begin diagnosing why that Delegate could not perform the task.

When you deploy a Workflow or Pipeline, you are taken to the Deployments page. On the Deployments page, look in the In the Location field in your browser and locate the Execution ID following executions. If you are deploying a Pipeline, the ID will follow pipeline-execution. For example, in the following string, rwXc5SXjQ86e3AscnxRedg is the execution ID:executions/rwXc5SXjQ86e3AscnxRedg/details

Step 3: Open the Delegate Log

The Delegate log is located in same folder as the Delegate and called delegate.log or delegate.<date>.log. You can open the log in a text editor or a log viewing program such as LogDNA. See Delegate Logs.

Step 4: Search for the Execution ID in the Log

For example, in LogDNA, put brackets around the Execution ID and search, like this: [NxO0JgbPSH-HL4TcN0zvaw] 

You will see log events for that Execution ID, indicated by executionId. For example, here is a log entry for Execution ID NxO0JgbPSH-HL4TcN0zvaw:

Jun 17 10:31:56 manager-79989476db-cz8k5 manager-Harness-Prod-KUBERNETES INFO [1.0.33801]
INFO software.wings.service.impl.instance.InstanceHelper - Handled deployment event for
executionId [NxO0JgbPSH-HL4TcN0zvaw],
infraMappingId [qR9_ifE0S9eLwt9zNa87Rw] of appId [fQ-liKwmS32DivKaF9Ggqg] successfully

Step 5: Find the Task ID

Task IDs are logged in many places in the log. We will use a task assignment example.

Once you have located the Execution ID records in the log, look at the records containing the uuid field. For example:

Jun 17 10:31:57 manager-79989476db-cz8k5 manager-Harness-Prod-KUBERNETES 
INFO [1.0.33801] [pool-12-thread-4749]
INFO [NxO0JgbPSH-HL4TcN0zvaw] software.wings.service.impl.DelegateServiceImpl -
Queueing async task: uuid: lRlt-zfPTj-vPMaA_mlF4w,
accountId: xxxxxxxxxxxx, type: K8S_COMMAND_TASK, correlationId: null

The value following uuid is the Task ID.

Step 6: Look for Queuing and Task submitted values using Task ID

For example:

Queueing async task: uuid: lRlt-zfPTj-vPMaA_mlF4w
Task submitted:...

Also, you will likely see the number of Delegates available to execute the task:

un 17 14:16:17 manager-79989476db-cz8k5 manager-Harness-Prod-KUBERNETES INFO [1.0.33801] [pool-12-thread-5063] INFO  [GrYOo5AgSBGHgp_h82ojig] software.wings.service.impl.DelegateServiceImpl - 8 delegates [yGpGxVhMSKaM_cn_2Bh4BQ, j_VbQsK2SDKx8URKFTKRZg, PoiAobskTXKinaqj-thIGQ, K0HJXUjgTn2v3kT09sbI3Q, _7K-s6zYQ26lRCss2t3ivw, 9krY9UruTVimTfBwFoKmnQ, btBGYgtCRjWIuZ1aC42Kog, BJquSItxStqmN9q59Mau5Q] eligible to execute task JIRA

Search using the Task ID to see what Delegate was used to execute the task:

Jun 17 15:23:39 manager-79989476db-vxlmq manager-Harness-Prod-KUBERNETES 
INFO [1.0.33801] [dw-88297 - PUT /api/delegates/TG01x2cRRJafGfgybNbYrg/tasks/gtnEdyxLSESVg1WYdItQtg/acquire?accountId=XEAgZ-j4RvirUgGObdd8-g]
INFO software.wings.service.impl.DelegateServiceImpl -
Task gtnEdyxLSESVg1WYdItQtg assigned to delegate TG01x2cRRJafGfgybNbYrg

Follow the log to see each step taken by the Delegate assigned that task.

Next Steps

How did we do?