8 - ECS Troubleshooting

Updated 3 weeks ago by Michael Cretzman

The following errors might occur when setting up and deploying ECS in Harness:

For information on ECS troubleshooting, see Amazon ECS Troubleshooting from AWS.

Rate Exceeded

A common issue with AWS deployments is exceeding an AWS rate limit for some AWS component, such as ECS clusters per region or maximum number of scaling policies per Auto Scaling Groups.

For steps to increase any AWS limits, see AWS Service Limits from AWS.

New ARN and Resource ID Format Must be Enabled

Harness uses tags for Blue/Green deployment, but ECS requires the new ARN and resource ID format be enabled to add tags to the ECS service.

If you have not opted into the new ECS ARN and resource ID format before you attempt Blue/Green deployment, you might receive the following error:

InvalidParameterException: The new ARN and resource ID format must be enabled to add tags to the service. Opt in to the new format and try again.

To solve this issue, opt into the new format and try again. For more information, see Migrating your Amazon ECS deployment to the new ARN and resource ID format from AWS.

Unable to Place a Task Because no Container Instance met all of its Requirements

The Upgrade Containers step might show the following message:

(service service-name) was unable to place a task because no container instance met all of its requirements.

Review the CPU requirements in both the task size and container definition parameters of the task definition.

See Service Event Messages from AWS.

Cannot Pull Container Image

You might see Docker errors indicating that when creating a task, the container image specified could not be retrieved.

See Cannot Pull Container Image Error from AWS.

Invalid CPU or Memory Value Specified

See the required settings in Invalid CPU or Memory Value Specified from AWS.

ClientException: Fargate requires that 'cpu' be defined at the task level

Ensure that you add the CPU and Memory settings in the Harness Service Container Specification section—for example:

"cpu" : "1",

"memory" : "512"

ClientException: The 'memory' setting for container is greater than for the task

In the Harness Service Container Specification JSON, there are two settings for memory. The memory setting for the container must not be greater than the memory setting for the task:

{

 "containerDefinitions" : [ {

   "name" : "${CONTAINER_NAME}",

   "image" : "${DOCKER_IMAGE_NAME}",

   "memory" : 512,

   ...

 } ],

 "executionRoleArn" : "${EXECUTION_ROLE}",

 ...

 "cpu" : "1",

 "memory" : "512",

 "networkMode" : "awsvpc"

}

Could Not Reach Http://<IP Address>:<Port>/V1/Tasks to Fetch Container Meta Data

The ECS container agent provides an API operation for gathering details about the container instance on which the agent is running and the associated tasks running on that instance. Harness uses the cURL command from within the container instance to query the Amazon ECS container agent on port 51678 and return container instance metadata or task information.

Ensure that the security group(s) used by AWS EC2 instances permits inbound traffic over TCP port 51678.

For more information, see Amazon ECS Container Agent Introspection from AWS.

AmazonElasticLoadBalancingException: Rate exceeded

You might receive this error as a result of AWS Load Balancer rate limiting. For more information, see Limits for Your Application Load Balancers and Limits for Your Classic Load Balancer from AWS.


How did we do?