Diagnose Delegate Issues
To diagnose Harness Delegate issues, you can use the general troubleshooting steps in this topic.
For information on general troubleshooting, see Troubleshooting.
To download or export deployment logs, which contain deployment events and errors, but not many Delegate events and errors, see Export Deployment Logs.
In this topic:
- Before You Begin
- Step 1: Look at Common Causes
- Step 2: Obtain Execution ID from the Deployment URL
- Step 3: Open the Delegate Log
- Step 4: Search for the Execution ID in the Log
- Step 5: Find the Task ID
- Step 6: Look for Queuing and Task submitted values using Task ID
- Next Steps
Before You Begin
- How Does Harness Manager Identify Delegates? and How Does Harness Manager Pick Delegates?
- Delegate Proxy Settings
- Delegate Logs
Step 1: Look at Common Causes
The following steps will help you diagnose many issues.
- Troubleshoot the Delegate validation URL. When a task is ready to be assigned, the Harness Manager first validates its lists of Delegates to see which Delegate should be assigned the task. It validates the Delegate using the URL in the task, such as a API call or SSH command. See How Does Harness Manager Pick Delegates?.
- Locate the URL by looking at the Delegate log.
- See if that URL is reachable from the Delegate host or from another host. Log into the host running the Delegate and see if the URL can be reached via cURL or ping.
- Ensure that the Delegate is not blacklisted. If a Delegate fails to perform a task that Delegate is blacklisted for that task and will not be tried again. TTL is 5 minutes. This is true if there is only one Delegate and even if the Delegate is Tagged for that task, such as with a Shell Script command in a Workflow.
- Use all Delegate Tags. If you are using Delegate Tags for a task (via Shell Script command) or Cloud Provider credentials, ensure that the command or provider has all of the Tags used by the Delegate. If the command or provider has less than all of the Tags used by the Delegate, the tagging will not work.
- Review Delegate Scoping. Ensure the Delegate is not scoped out of performing the task. See Delegate Scope and Best Practices and Notes.
- Delegate Restarting. Check to see if the Delegate is restarting (for example, because the socket connection is failing) and offline at the time of the task.
Step 2: Obtain Execution ID from the Deployment URL
You can diagnose a lot of Delegate issues using the Execution ID, Task ID and Delegate information from the task assignment and operation.
A deployment has an Execution ID. Every Execution ID includes one or more tasks, identified by Task IDs. Using the Task ID, you can determine which Delegate was assigned the task and begin diagnosing why that Delegate could not perform the task.
When you deploy a Workflow or Pipeline, you are taken to the Deployments page. On the Deployments page, look in the In the Location field in your browser and locate the Execution ID following executions. If you are deploying a Pipeline, the ID will follow pipeline-execution. For example, in the following string, rwXc5SXjQ86e3AscnxRedg is the execution ID:
Step 3: Open the Delegate Log
The Delegate log is located in same folder as the Delegate and called delegate.log or delegate.<date>.log. You can open the log in a text editor or a log viewing program such as LogDNA. See Delegate Logs.
Step 4: Search for the Execution ID in the Log
For example, in LogDNA, put brackets around the Execution ID and search, like this:
You will see log events for that Execution ID, indicated by
executionId. For example, here is a log entry for Execution ID
Jun 17 10:31:56 manager-79989476db-cz8k5 manager-Harness-Prod-KUBERNETES INFO [1.0.33801]
INFO software.wings.service.impl.instance.InstanceHelper - Handled deployment event for
infraMappingId [qR9_ifE0S9eLwt9zNa87Rw] of appId [fQ-liKwmS32DivKaF9Ggqg] successfully
Step 5: Find the Task ID
Task IDs are logged in many places in the log. We will use a task assignment example.
Once you have located the Execution ID records in the log, look at the records containing the uuid field. For example:
Jun 17 10:31:57 manager-79989476db-cz8k5 manager-Harness-Prod-KUBERNETES
INFO [1.0.33801] [pool-12-thread-4749]
INFO [NxO0JgbPSH-HL4TcN0zvaw] software.wings.service.impl.DelegateServiceImpl -
Queueing async task: uuid: lRlt-zfPTj-vPMaA_mlF4w,
accountId: xxxxxxxxxxxx, type: K8S_COMMAND_TASK, correlationId: null
The value following
uuid is the Task ID.
Step 6: Look for Queuing and Task submitted values using Task ID
Queueing async task: uuid: lRlt-zfPTj-vPMaA_mlF4w
Also, you will likely see the number of Delegates available to execute the task:
un 17 14:16:17 manager-79989476db-cz8k5 manager-Harness-Prod-KUBERNETES INFO [1.0.33801] [pool-12-thread-5063] INFO [GrYOo5AgSBGHgp_h82ojig] software.wings.service.impl.DelegateServiceImpl - 8 delegates [yGpGxVhMSKaM_cn_2Bh4BQ, j_VbQsK2SDKx8URKFTKRZg, PoiAobskTXKinaqj-thIGQ, K0HJXUjgTn2v3kT09sbI3Q, _7K-s6zYQ26lRCss2t3ivw, 9krY9UruTVimTfBwFoKmnQ, btBGYgtCRjWIuZ1aC42Kog, BJquSItxStqmN9q59Mau5Q] eligible to execute task JIRA
Search using the Task ID to see what Delegate was used to execute the task:
Jun 17 15:23:39 manager-79989476db-vxlmq manager-Harness-Prod-KUBERNETES
INFO [1.0.33801] [dw-88297 - PUT /api/delegates/TG01x2cRRJafGfgybNbYrg/tasks/gtnEdyxLSESVg1WYdItQtg/acquire?accountId=XEAgZ-j4RvirUgGObdd8-g]
INFO software.wings.service.impl.DelegateServiceImpl -
Task gtnEdyxLSESVg1WYdItQtg assigned to delegate TG01x2cRRJafGfgybNbYrg
Follow the log to see each step taken by the Delegate assigned that task.