Background
Testing VPC:
2 private subnets with NACLS that allow all inbound from IPs within the VPC and all outbound traffic. The subnets have a route table configured to a NAT gateway within a public subnet.
2 public subnets that allow all inbound/outbound traffic. One of the subnets contains the NAT gateway and both subnets have a route table pointing to the Internet Gateway.
Problem
When running an ECS Fargate task (platform: 1.4) within one of the private subnets, the following error arises:
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.us-west-2.amazonaws.com/: dial tcp <IP>:443: i/o timeout
The ECS task contains one container that uses a private ECR image hosted within the same AWS account. The security group associated with the task allows all inbound traffic from IPs within the VPC and allows all outbound traffic.
ECS task execution role contains the following policies:
"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
"arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
Attempts:
When the private subnets NACL's inbound access was changed to allow all types of traffic, the ECS task was strangely able to pull the ECR image.
Created the VPC endpoints mentioned in this article with the correct security groups but got the same error.
I'm tempted to try following this guide although it specifically says:
If your task definition references an image that's stored in Amazon ECR, this topic doesn't apply.
I assume you are pulling an image from ECR?
If you are launching via ecs-cli, add this to the ecs-params.yml
(read about that file here: https://github.com/aws/amazon-ecs-cli)
Firstly: auto-assign public IP addresses ->
run_params:
network_configuration:
awsvpc_configuration:
subnets:
- "subnet-06786b976xx"
- "subnet-0b9dxxxxxxx"
security_groups:
- "sg-08157xxxxxxxxx4"
assign_public_ip: "ENABLED"
Secondly make sure that the security groups you use for your VPC allow 80/443 traffic
Thirdly make sure in AWS IMA that the execution role (ecsTaskExecutionRole) has policies including:
AWSAppRunnerServicePolicyForECRAccess
Related
I have created an ECS service and started a service with a task definition.
I wan't the ability to start these services in a private subnet, but I read that for that to happen I need to use:
NAT Gateway
or
VPC Endpoints for S3 and ECR (API and DKR). Optional are Secret Manager and CLoudWatch
I don't want to to pay fir NAT Gateway so I started configuring the option with VPC endpoints.
I created:
ECR DKR and API Interface endpoints targeting the subnets that I use
S3 gateway endpoint attached to the route table of the subnet I am using
Secret Manager Interface endpoint (For future application using secrets)
CloudWatch interface endpoint for logging
I also attached the interface endpoints to a security group that has:
Ingress: 80 and 443 - 0.0.0.0/0
egress: All
enabled DNS support, hostnames and resolution on the VPC.
And created the correct ECS task execution and task role for the service with ecr and cloud watch permissions.
But regardless all of this configuration, I am getting this error when starting up the service:
Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.eu-central-1.amazonaws.com/": dial tcp 52.119.188.80:443: i/o timeout
I don't know already what to do, I freaking out, I am trying to get the resolution for already few hours.
If someone knows how to fix it, I would appreciate that.
Thanks
I have an architecture similar to https://github.com/aws-samples/ecs-refarch-cloudformation
I would like to know if I can have an AutoScalingGroup with instances in private subnets without using NAT Gateway?
I was experimenting with removing the NATs and adding VPC endpoints but I always end up with problem like this:
2022-08-21 10:55:07 UTC+1000 <MY_ECS_TEMPLATE> CREATE_FAILED The following resource(s) failed to create: [ECSAutoScalingGroup].
2022-08-21 10:55:07 UTC+1000 ECSAutoScalingGroup CREATE_FAILED Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement
if I can have an AutoScalingGroup with instances in private subnets without using NAT Gateway?
Sadly no. But you can create a vpc interface endpoints for ecs which will enable communication with ecs from your private subnets, without the need for internet and nat.
But the error can be also from cloudformation (CFN) which can't get information back if ec2 instanses launched correctly. So you may need to add vpc endpoints for CFN as well.
I had similar issue. We use AWS Batch, which is based on AutoScaling Groups. We use squid proxy for internet access from private subnets. After I created few endpoints as mentioned before, spot VM started. Next I had to update AMI for these VM and set proper proxy/noproxy. And it works.
I have created an ECS Fargate Task, which I can manually run. It updates a Dynomodb and I get logs.
Now I want this to run on a schedule. I have setup a scheduled ECS task through EventBridge. However, this does not run.
My looking at the EventBridge logs I can see that the container has been stopped for the following stopped reason:
ResourceInitializationError: unable to pull secrets or registry auth: execution resource
retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3
time(s): RequestError: send request failed caused by: Post https://api.ecr....
I thought this might be a problem with permissions. However, I tested giving the Task Execution Role full power user permissions and I still get the same error. Could the problem be something else?
This is due to a connectivity issue.
The docs say the following:
For tasks on Fargate, in order for the task to pull the container image it must either use a public subnet and be assigned a public IP address or a private subnet that has a route to the internet or a NAT gateway that can route requests to the internet.
So you need to make sure your task has a route to an internet gateway (i.e. it's in a Public subnet) or a NAT gateway.
Alternatively, if your service is in an isolated subnet, you need to create VPC endpoints for ECR and other services you need to call, as described in the docs:
To allow your tasks to pull private images from Amazon ECR, you must create the interface VPC endpoints for Amazon ECR.
When you create a scheduled task, you also specify the networking options. The docs mention this step:
(Optional) Expand Configure network configuration to specify a network configuration. This is required for tasks hosted on Fargate and for tasks using the awsvpc network mode.
For Subnets, specify one or more subnet IDs.
For Security groups, specify one or more security group IDs.
For Auto-assign public IP, specify whether to assign a public IP address from your subnet to the task.
So the networking configuration changed between the manually run task and the scheduled task. Refer to the above to figure out the needed settings for your case.
I fixed this by enabling auto-assign public IP.
However, to do this, I had to first change from "Capacity provider strategy" -
"Use cluster default", to "Launch type" - "FARGATE". Then the option to enable auto-assign public IP became available in the dropdown in the EventBridge UI.
This seems odd to me, because my default capacity provider strategy for my cluster is Fargate. But it is working now.
Need to use a gateway to follow the traffic from ECS to ECR. It can either Internet Gateway or NAT Gateway eventually which would be effecting cost factor.
But where we can resolve this scenario, by creating VPC Endpoints. Which maintains the traffic within the AWS Resources.
Endpoints Required for this would be :
S3 Gateway
ECR
ECS
I am trying to use CodeDeploy to deploy my revisions on private subnet instances using VPC endpoint.
VPC endpoint is having required subnet configured and security group applied on VPC endpoint is having all inbound and outbound allowed. Still my deployment is failing and I do not have any way to check logs on EC2 instance since it is private. Any help or guidance regarding this?
I am trying to follow the below link:
https://aws.amazon.com/about-aws/whats-new/2020/08/aws-codedeploy-now-supports-deployments-to-virtual-private-cloud-endpoints/
I am using docker containers with secrets on ECS, without problems. After moving to fargate and platform 1.4 for efs support i start getting the following error.
Any help please?
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 1 time(s): secret arn:aws:secretsmanager:eu-central-1:.....
Here's a checklist:
If your ECS tasks are in a public subnet (0.0.0.0/0 routes to Internet Gateway) make sure your tasks can call the "public" endpoint for Secrets Manager. Basically, outbound TCP/443.
If your ECS tasks are in a private subnet, make sure that one of the following is true: (a) your instances need to connect to the Internet through a NAT gateway (0.0.0.0/0 routes to NAT gateway) or (b) you have an AWS PrivateLink endpoint to secrets manager connected to your VPC (and to your subnets)
If you have an AWS PrivateLink connection, make sure the associated Security Group has inbound access from the security groups linked to your ECS tasks.
Make sure you have set GetSecretValue IAM permission to the ARN(s) of the secrets manager entry(or entries) set in the ECS "tasks role".
Edit: Here's another excellent answer - https://stackoverflow.com/a/66802973
I had the same error message, but the checklist above misses the cause of my problem. If you are using VPC endpoints to access AWS services (ie, secretsmanager, ecr, SQS, etc) then those endpoints MUST permit access to the security group that is associated with the VPC subnet that your ECS instance is running in.
Another watchit is, if you are using EFS to host volumes, ensure that your volumes can be mounted by the same security group identified above. Go to EFS, select the appropriate file system, Network tab, then Manage.