Can't push Dockerimages to ECR - amazon-ecr

I get an error on push my local Dockerimage to my private ECR:
My IAM-User has AmazonEC2ContainerRegistryFullAccess rights and my EC2 too.
$ aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin xx.dkr.ecr.eu-central-1.amazonaws.com
...
Login Succeeded
$ aws ecr describe-repositories
{
"repositories": [
{
"repositoryUri": "xx.dkr.ecr.eu-central-1.amazonaws.com/my_repo",
"imageScanningConfiguration": {
"scanOnPush": false
},
"encryptionConfiguration": {
"encryptionType": "AES256"
},
"registryId": "xx",
"imageTagMutability": "MUTABLE",
"repositoryArn": "arn:aws:ecr:eu-central-1:xx:repository/my_repo",
"repositoryName": "my_repo",
"createdAt": 1650817284.0
}
]
}
$ docker pull hello-world
$ docker tag hello-world:latest xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world:latest
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world latest feb5d9fea6a5 7 months ago 13.3kB
hello-world latest feb5d9fea6a5 7 months ago 13.3kB
and now i get the error on push my image:
$ docker push xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world:latest
The push refers to repository [xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world]
e07ee1baac5f: Retrying in 1 second
EOF
Any suggestions?
The profile-trick from https://stackoverflow.com/a/70453287/10243980 works NOT.
Many thanks

One of my working example is the following
aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.eu-central-1.amazonaws.com
docker build -t dolibarr .
docker tag dolibarr:latest 123456789012.dkr.ecr.eu-central-1.amazonaws.com/dolibarr:latest
docker push 123456789012.dkr.ecr.eu-central-1.amazonaws.com/dolibarr:latest
Compared to your commands, it looks very similar. So now, please check, if your user is able to push to the repository itself (ecr:PutImage). Probably this is the main issue.
A good solution to find more help is the following Pushing an image to ECR, getting "Retrying in ... seconds"
My policy for my Docker image role, I am using, is the following (terraform style):
{
Action = [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart",
]
Effect = "Allow"
Resource = "*"
}
Try to adjust your policy and remove the "Principal" entry. This is not necessary.
Another possible reason could has nothing to do with the policy:
Do you use some local proxy? I experienced some issues with using Proxy Servers for all public endpoints, like ECR, S3, etc. I disabled to use for those domains and it worked (depends on using VPN, or something similar).

You need to create a repository with the name hello-world. It is explained at the begining of Pushing a Docker image ecr docs.

Related

How do I get a Docker Swarm manager to pull images from AWS ECR using IAM Role permissions?

I'm having trouble pulling images from AWS ECR, running Docker Swarm. It's been working ok for years, but my swarm manager nodes were changed to new EC2 instances. Now my services fail to deploy:
~ $ docker stack deploy -c dkr_compose_geo_site:3.2.0 --with-registry-auth geo_stack
The manager node log shows "no basic auth credentials":
May 19 21:21:12 ip-172-31-3-108 root: time="2020-05-19T21:21:12.857007050Z" level=error msg="pulling image failed" error="Get https://445523.dkr.ecr.us-west-2.amazonaws.com/v2/geo_site/manifests/sha256:da5820742cd0ecd52e3a2c61179a039ce80996564604b70465e3966087380a09: no basic auth credentials" module=node/agent/taskmanager node.id=eix8c6orbunemismg03ib1rih service.id=smilb788pets7y5rgbu3aze9l task.id=zd3ozdpr9exphwlz318pa9lpe
May 19 21:21:12 ip-172-31-3-108 root: time="2020-05-19T21:21:12.857701347Z" level=error msg="fatal task error" error="No such image: 445523.dkr.ecr.us-west-2.amazonaws.com/geo_site#sha256:da5820742cd0ecd52e3a2c61179a039ce80996564604b70465e3966087380a09" module=node/agent/taskmanager node.id=eix8c6orbunemismg03ib1rih service.id=smilb788pets7y5rgbu3aze9l task.id=zd3ozdpr9exphwlz318pa9lpe
This manager node is running on an EC2 Instance with an IAM Role; the IAM Role has an ECR policy that appears to grant permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
From reading the AWS/Docker docs, I thought docker commands run on a manager node should adopt the Instance IAM Role and access the ECR repo using the associated policy permissions. It's always seemed to work that way, but now it's looking like there might have been some config file hidden on the old manager node; I'm on a new instance and it doesn't work. I don't run an AWS-CLI on these manager nodes, so there's no aws ecr get-login to login manually. How do I get this new manager node to authenticate with ECR?
Thanks!
My solution, based on comment by Luigi Lopez and amazon-ecr-credential-helper:
The AWS IAM Role allows authentication, but the docker cli must still present credentials to the ECR, as Luigi pointed out in his comment.
This is a Docker Swarm implementation, with nodes running the Alpine OS. There is an aws-cli package available for Alpine, but the installation took a lot of fussing around and in the end the binary crashed anyway.
The Amazon ECR Credential Helper is a better long-term solution in any case because you don't need to get new tokens every 12 hours or set up a proxy server, etc. It uses the recommended IAM Role authentication, with no credentials stored on the machine or leaking into log files.
So under Alpine I followed the instructions in the link above to build from sources.
I installed go, git, and make, and then built the credential-helper as described. I set up the PATH as described, created a config file, and then my deployment worked. There's no docker login required.

Terminology - ECR repo uri vs ECR repo name

In the below json received after talking to AWS ECR end point service:
{
"repository": {
"repositoryArn": "arn:aws:ecr:us-west-2:11122233334444:repository/some_app_image",
"registryId": "11122233334444",
"repositoryName": "some_app_image",
"repositoryUri": "11122233334444.dkr.ecr.us-west-2.amazonaws.com/some_app_image",
"createdAt": 11111111554.0,
"imageTagMutability": "MUTABLE",
"imageScanningConfiguration": {
"scanOnPush": false
}
}
}
after running command: aws ecr describe-repositories --repository-names some_app_image
How to term 11122233334444.dkr.ecr.us-west-2.amazonaws.com? Is it an ECR end point?
You would refer to it as your registry URL. More information on terminology at the ECR user docs
The value in repositoryUri is what you would use in a command like docker pull. So in this example you would say docker pull 11122233334444.dkr.ecr.us-west-2.amazonaws.com/some_app_image to download your image.

Spinnaker + ECR access

I'm having trouble setting up Spinnaker with ECR access.
Background: I installed spinnaker using helm on an EKS cluster and I've confirmed that the cluster has the necessary ECR permissions (by manually running ECR commands from within the clouddriver pod). I am following the instructions here to get Spinnaker+ECR set up: https://www.spinnaker.io/setup/install/providers/docker-registry/
Issue: When I run:
hal config provider docker-registry account add my-ecr-registry \
--address $ADDRESS \
--username AWS \
--password-command "aws --region us-west-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | sed 's/^AWS://'"
I get the following output:
+ Get current deployment
Success
- Add the some-ecr-registry account
Failure
Problems in default.provider.dockerRegistry.some-ecr-registry:
- WARNING Resolved Password was empty, missing dependencies for
running password command?
- WARNING You have a supplied a username but no password.
! ERROR Unable to fetch tags from the docker repository: code, 400
Bad Request
? Can the provided user access this repository?
- WARNING None of your supplied repositories contain any tags.
Spinnaker will not be able to deploy any docker images.
? Push some images to your registry.
Problems in halconfig:
- WARNING There is a newer version of Halyard available (1.28.0),
please update when possible
? Run 'sudo apt-get update && sudo apt-get install
spinnaker-halyard -y' to upgrade
- Failed to add account some-ecr-registry for provider
dockerRegistry.
I have confirmed that the aws-cli is installed on the clouddriver pod. And I've confirmed that I can the password-command directly from the clouddriver pod and it successfully returns a token.
I've also confirmed that if I manually generate an ECR token and run hal config provider docker-registry account add my-ecr-registry --address $ADDRESS --username AWS --password-command "echo $MANUALLY_GENERATED_TOKEN" everything works fine. So there is something specific to the password-command that is going wrong and I'm not sure how to debug this.
One other odd behavior: if I simplify the password command to be: hal config provider docker-registry account add some-ecr-registry --address $ADDRESS --username AWS --repositories code --password-command "aws --region us-west-2 ecr get-authorization-token" , I get an addt'l piece of output that says "- WARNING Password command returned non 0 return code stderr/stdout was:bash: aws: command not found". This output only appears for this simplified command.
Any advice on how to debug this would be much appreciated.
If like me your ECR registry is in another account, then you have to forcibly assume the role for the target account where your registry resides
passwordCommand: read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< `aws sts assume-role --role-arn arn:aws:iam::<AWS_ACCOUNT>:role/<SPINNAKER ROLE_NAME> --query "[Credentials.AccessKeyId, Credentials.SecretAccessKey, Credentials.SessionToken]" --output text --role-session-name spinnakerManaged-w2`; export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN; aws ecr get-authorization-token --region us-west-2 --output text --query 'authorizationData[].authorizationToken' --registry-ids <AWS_ACCOUNT> | base64 -d | sed 's/^AWS://'
Credits to https://github.com/spinnaker/spinnaker/issues/5374#issuecomment-607468678
I also installed Spinnaker on AKS and all i did was by using an AWS Managing User with the correct AWS IAM policy to ECR:* i have access to the ECR repositories directly.
I dont think that hal being java based will execute the Bash command in --password-command
set the AWS ECS provider in your spinnaker deployment
Use the Following AWS IAM policy (SpinnakerManagingPolicy) to be attached to the AWS MAnaging User to give access to ECR. Please replace the AWS Accounts based on your need.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:*",
"cloudformation:*",
"ecr:*"
],
"Resource": [
"*"
]
},
{
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::123456789012:role/SpinnakerManagedRoleAccount1",
"arn:aws:iam::101121314157:role/SpinnakerManagedRoleAccount2",
"arn:aws:iam::202122232425:role/SpinnakerManagedRoleAccount3"
],
"Effect": "Allow"
}
]
}

AWS ElasticBeanstalk pull Docker image from Gitlab registry

I’m having a hard time to pull Docker image from private Gitlab registry to AWS MultiContainer ElasticBeanstalk environment.
I have added .dockercfg into S3 in the same region as my cluster and also allowed to aws-elasticbeanstalk-ec2-role IAM role to get data from S3.
ElasticBeanstalk always return error CannotPullContainerError: API error (500)
My .dockercfg is in this format:
{
"https://registry.gitlab.com" : {
"auth" : “my gitlab deploy token“,
"email" : “my gitlab token name“
}
}
Inside Dockerrun.aws.json I have added following
"authentication": {
"bucket": "name of my bucket",
"key": ".dockercfg"
},
When I try to login via docker login -u gitlabtoken-name -p token it works perfectly.
The gitlab deploy token is not the auth key.
To generate a proper auth key I usually do the following:
docker run -ti docker:dind sh -c "docker login -u name -p deploy-token registry.gitlab.com && cat /root/.docker/config.json"
and it'll print something like:
{
"auths": {
"registry.gitlab.com": {
"auth": "your-auth-key"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.09.0 (linux)"
}
}
Then, as per elasticbeanstalk docs "Using Images From a Private Repository
", you should take just what it's needed.
Hope this'll help you!

How to make 'aws ecr get-login' across regions?

I have a docker registry in AWS ECR in region 'us-east-1'. Everything works fine on EC2 instances launched in 'us-east-1'. But when I launch an instance in 'eu-central-1' and try to run
$(aws ecr get-login --region us-east-1)
I get the following response
Error response from daemon: Get https://acc-id.dkr.ecr.us-east-1.amazonaws.com/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
If I run
aws ecr get-login --region us-east-1
I do see the following response
docker login -u AWS -p xxxx -e none https://acc_id.dkr.ecr.us-east-1.amazonaws.com
Ec2 instance has the following policy for the iam-role:
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
"Resource": "*"
}
Please tell me how can I have this cross-region ECR accessibility.
Below procedure can be used for cross region image pull from ECR:
$(aws ecr get-login --no-include-email --region <region having repository> --registry-ids <id>)
docker pull <id>.dkr.ecr.us-west-1.amazonaws.com/<image_name>:<tag>
Below sample shows instance in region us-east-1 is pulling AWS Deep Learning Container Image from ECR hosted in us-west-1
To get region from EC2 instance metadata:
curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | grep region
"region" : "us-east-1"
ECR Login:
$(aws ecr get-login --no-include-email --region us-west-1 --registry-ids 763104351884)
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Docker Pull:
docker pull 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04
1.13-cpu-py27-ubuntu16.04: Pulling from tensorflow-training
34667c7e4631: Already exists
d18d76a881a4: Already exists
119c7358fbfc: Already exists
2aaf13f3eff0: Already exists
7b890657bd19: Already exists
f095a52e6583: Already exists
182b3abfb706: Already exists
89cc0ffab23c: Already exists
7a24716f4857: Already exists
Digest: sha256:65c7f58ac49ed39d5b6bde4f3800dbbf8d9f99b5316292b27315fb6a4b8be56b
Status: Downloaded newer image for 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04