ECR image push AWS CodeBuild issue - amazon-web-services

COMMAND_EXECUTION_ERROR: Error while executing command: $(aws ecr get-login --no-include-email --region us-east-1). Reason: exit status 127
Below is my buildspec.yml file
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- $(aws ecr get-login --region ***-east-*)
- REPOSITORY_URI=***********.dkr.ecr.***-east-*.amazonaws.com/repositoryname
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION)
- IMAGE_TAG=${COMMIT_HASH:=latest}
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
- echo Writing definitions file...
- printf '[{"name":"project-container","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > taskdefinition.json
artifacts:
files: taskdefinition.json

In case it helps someone else, for the work I'm doing inside my build scripts executed by CodeBuild. These are the IAM permissions I had to add (finding them one by one as I hit the error).
{
"Action": [
"ecr:GetAuthorizationToken",
"ecr:DescribeRepositories",
"ecr:CreateRepository",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:BatchCheckLayerAvailability",
"ecr:PutImage",
"ecs:UpdateService"
],
"Resource": "*",
"Effect": "Allow"
} '
I'm sure there are more permissions that may be required if you're doing stuff I'm not doing in your builds. I'm pushing to ECR and forcing the Service (and the related tasks) to deploy the new image.

Your post has inconsistent details, is that intentional? If not, it may be causing the problem. Your code snippet says:
$(aws ecr get-login --region ***-east-*)
Perhaps you purposely redacted the region (what's the point of that btw?) but why is it missing the --no-include-email? Higher up in your post, you do make reference to --no-include-email, so I know you're aware of it.
Run the process outside of a subshell to get a better log
Rather than running it inside a subshell (e.g. $(my command)), for troubleshooting purposes, try running taking the subshell out so you can get better output. Report the results here so we can troubleshoot the error you get.
aws ecr get-login --no-include-email --region us-east-1 <- try this temporarily
vs.
$(aws ecr get-login --no-include-email --region us-east-1)
Have you created an IAM Policy with ECR permissions for CodeBuild to use?
This is very important. CodeBuild needs permission to access ECR on your behalf. Here's an example I found on this blog article. It may need tweaking to your needs. http://beta.awsdocs.com/services/code_build/build_docker_images/
{
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"
],
"Resource": "*",
"Effect": "Allow"
}

Related

Can't push Dockerimages to ECR

I get an error on push my local Dockerimage to my private ECR:
My IAM-User has AmazonEC2ContainerRegistryFullAccess rights and my EC2 too.
$ aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin xx.dkr.ecr.eu-central-1.amazonaws.com
...
Login Succeeded
$ aws ecr describe-repositories
{
"repositories": [
{
"repositoryUri": "xx.dkr.ecr.eu-central-1.amazonaws.com/my_repo",
"imageScanningConfiguration": {
"scanOnPush": false
},
"encryptionConfiguration": {
"encryptionType": "AES256"
},
"registryId": "xx",
"imageTagMutability": "MUTABLE",
"repositoryArn": "arn:aws:ecr:eu-central-1:xx:repository/my_repo",
"repositoryName": "my_repo",
"createdAt": 1650817284.0
}
]
}
$ docker pull hello-world
$ docker tag hello-world:latest xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world:latest
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world latest feb5d9fea6a5 7 months ago 13.3kB
hello-world latest feb5d9fea6a5 7 months ago 13.3kB
and now i get the error on push my image:
$ docker push xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world:latest
The push refers to repository [xx.dkr.ecr.eu-central-1.amazonaws.com/hello-world]
e07ee1baac5f: Retrying in 1 second
EOF
Any suggestions?
The profile-trick from https://stackoverflow.com/a/70453287/10243980 works NOT.
Many thanks
One of my working example is the following
aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.eu-central-1.amazonaws.com
docker build -t dolibarr .
docker tag dolibarr:latest 123456789012.dkr.ecr.eu-central-1.amazonaws.com/dolibarr:latest
docker push 123456789012.dkr.ecr.eu-central-1.amazonaws.com/dolibarr:latest
Compared to your commands, it looks very similar. So now, please check, if your user is able to push to the repository itself (ecr:PutImage). Probably this is the main issue.
A good solution to find more help is the following Pushing an image to ECR, getting "Retrying in ... seconds"
My policy for my Docker image role, I am using, is the following (terraform style):
{
Action = [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart",
]
Effect = "Allow"
Resource = "*"
}
Try to adjust your policy and remove the "Principal" entry. This is not necessary.
Another possible reason could has nothing to do with the policy:
Do you use some local proxy? I experienced some issues with using Proxy Servers for all public endpoints, like ECR, S3, etc. I disabled to use for those domains and it worked (depends on using VPN, or something similar).
You need to create a repository with the name hello-world. It is explained at the begining of Pushing a Docker image ecr docs.

How to authenticate AWS EBS (Beanstalk) on ECS (Container Storage)? AccessDeniedException

I'm new to using AWS EBS and ECS, so please forgive me if I ask a question which might be obvious for others. To the issue:
I've configured EBS and ECS in the same AWS cloud region. I've pushed my container and can see the image in ECS. The EBS environment has been built using a Dockerrun.aws.json pointing to the ECS. During the build of the EBS I get the following error:
2020/11/02 20:50:25.858567 [INFO] authenticate with ECR if the image is in an ECR repo
2020/11/02 20:50:25.858582 [INFO] Running command /bin/sh -c aws ecr get-login --no-include-email --registry-ids 731178912345 --region eu-west-1
2020/11/02 20:50:28.355911 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to authenticate with ECR for registry 731178912XXX in eu-west-1: Command /bin/sh -c aws ecr get-login --no-include-email --registry-ids 731178912XXX --region eu-west-1 failed with error exit status 255. Stderr:
An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts::731178912XXX:assumed-role/aws-elasticbeanstalk-ec2-role/i-0bff9b5324348ea71 is not authorized to perform: ecr:GetAuthorizationToken on resource: *
I read somewhere that the identify is automatically passed in to process the authentication. I couldn't find any indication on what to do to resolve the issue. Maybe someone could help me out?
After adding the permission as discussed I get these:
2020/11/03 09:17:46.490399 [INFO] pull docker image if update is not false in Dockerrun.aws.json
2020/11/03 09:17:46.490431 [INFO] Running command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest
2020/11/03 09:17:46.576648 [WARN] failed to execute command: docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest, retrying...
2020/11/03 09:17:46.576673 [INFO] Running command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest
2020/11/03 09:17:46.659379 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to pull docker image: Command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest failed with error exit status 1. Stderr:Error response from daemon: pull access denied for 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project, repository does not exist or may require 'docker login': denied: User: arn:aws:sts::731178912XXX:assumed-role/aws-elasticbeanstalk-ec2-role/i-0aa453e0d44cdfa90 is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:eu-west-1:731178912XXX:repository/user/project
Your aws-elasticbeanstalk-ec2-role instance role does not have permissions to execute ecr:GetAuthorizationToken.
You can address this, by adding an inline policy to the aws-elasticbeanstalk-ec2-role role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
}
]
}
Please note that more permissions may be needed after this one. You can add them in the same way.

Spinnaker + ECR access

I'm having trouble setting up Spinnaker with ECR access.
Background: I installed spinnaker using helm on an EKS cluster and I've confirmed that the cluster has the necessary ECR permissions (by manually running ECR commands from within the clouddriver pod). I am following the instructions here to get Spinnaker+ECR set up: https://www.spinnaker.io/setup/install/providers/docker-registry/
Issue: When I run:
hal config provider docker-registry account add my-ecr-registry \
--address $ADDRESS \
--username AWS \
--password-command "aws --region us-west-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | sed 's/^AWS://'"
I get the following output:
+ Get current deployment
Success
- Add the some-ecr-registry account
Failure
Problems in default.provider.dockerRegistry.some-ecr-registry:
- WARNING Resolved Password was empty, missing dependencies for
running password command?
- WARNING You have a supplied a username but no password.
! ERROR Unable to fetch tags from the docker repository: code, 400
Bad Request
? Can the provided user access this repository?
- WARNING None of your supplied repositories contain any tags.
Spinnaker will not be able to deploy any docker images.
? Push some images to your registry.
Problems in halconfig:
- WARNING There is a newer version of Halyard available (1.28.0),
please update when possible
? Run 'sudo apt-get update && sudo apt-get install
spinnaker-halyard -y' to upgrade
- Failed to add account some-ecr-registry for provider
dockerRegistry.
I have confirmed that the aws-cli is installed on the clouddriver pod. And I've confirmed that I can the password-command directly from the clouddriver pod and it successfully returns a token.
I've also confirmed that if I manually generate an ECR token and run hal config provider docker-registry account add my-ecr-registry --address $ADDRESS --username AWS --password-command "echo $MANUALLY_GENERATED_TOKEN" everything works fine. So there is something specific to the password-command that is going wrong and I'm not sure how to debug this.
One other odd behavior: if I simplify the password command to be: hal config provider docker-registry account add some-ecr-registry --address $ADDRESS --username AWS --repositories code --password-command "aws --region us-west-2 ecr get-authorization-token" , I get an addt'l piece of output that says "- WARNING Password command returned non 0 return code stderr/stdout was:bash: aws: command not found". This output only appears for this simplified command.
Any advice on how to debug this would be much appreciated.
If like me your ECR registry is in another account, then you have to forcibly assume the role for the target account where your registry resides
passwordCommand: read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< `aws sts assume-role --role-arn arn:aws:iam::<AWS_ACCOUNT>:role/<SPINNAKER ROLE_NAME> --query "[Credentials.AccessKeyId, Credentials.SecretAccessKey, Credentials.SessionToken]" --output text --role-session-name spinnakerManaged-w2`; export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN; aws ecr get-authorization-token --region us-west-2 --output text --query 'authorizationData[].authorizationToken' --registry-ids <AWS_ACCOUNT> | base64 -d | sed 's/^AWS://'
Credits to https://github.com/spinnaker/spinnaker/issues/5374#issuecomment-607468678
I also installed Spinnaker on AKS and all i did was by using an AWS Managing User with the correct AWS IAM policy to ECR:* i have access to the ECR repositories directly.
I dont think that hal being java based will execute the Bash command in --password-command
set the AWS ECS provider in your spinnaker deployment
Use the Following AWS IAM policy (SpinnakerManagingPolicy) to be attached to the AWS MAnaging User to give access to ECR. Please replace the AWS Accounts based on your need.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:*",
"cloudformation:*",
"ecr:*"
],
"Resource": [
"*"
]
},
{
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::123456789012:role/SpinnakerManagedRoleAccount1",
"arn:aws:iam::101121314157:role/SpinnakerManagedRoleAccount2",
"arn:aws:iam::202122232425:role/SpinnakerManagedRoleAccount3"
],
"Effect": "Allow"
}
]
}

CREATE_FAILED while creating the CloudFormation Stack from Hyperledger template

I am using this template to create the stack:
https://aws-blockchain-templates-us-east-1.s3.us-east-1.amazonaws.com/hyperledger/fabric/templates/simplenetwork/latest/hyperledger.template.yaml
While following this blog-post from AWS, I am getting an error.
Blog - Post Link :
https://aws.amazon.com/blockchain/templates/getting-started/
Region : us-east-1
Error Message :
The following resource(s) failed to create: [FabricEC2CommonStack]. . Rollback requested by user.
CREATE_FAILED AWS::CloudFormation::Stack FabricEC2CommonStack Embedded stack arn:aws:cloudformation:us-east-1:>:stack/FabricStack-FabricEC2CommonStack-NNFUD6RJCZB1/<> was not successfully created: The following resource(s) failed to create: [EC2InstanceForDev].
I have met all the prerequisites.
What could be the reason for this error and how to rectify it?
After this, I get ROLLBACK_IN_PROGRESS and ROLLBACK_COMPLETE.
The Official AWS Blockchain Cloud Formation Template for Hyperledger Fabric is a nested template (our base template calls another template which does all the setup on an EC2 instance which itself creates).
But the problem is it does everything on the EC2-Instance except installing docker-compose & it throws an error that docker-compose command not found at the end which causes the CloudFormation template to break(EC2InstanceForDev) and do a rollback. So instead of using CloudFormation Template, we can run the same script manually on the EC2-instance with a small change. The change is to install docker-compose beforehand. Rest setup remains the same i.e -- 1. Create a VPC, 2. Create Public Subnets, 3. Create EIP if you want to attach it later, 4. Create Key-Pair for SSH, 5. Create IAM Role & Policy, 6. Create Security Group with Inbound 8080(TCP) & 22(SSH), 7. launch an EC2 Instance with the created resources in step (1to6).
AMI which is preferred is -
ami-1853ac65 for us-east-1
ami-25615740 for us-east-2
ami-dff017b8 for us-west-2
Docker Image Repository -
354658284331 for us-east-1
763976151875 for us-east-2
712425161857 for us-west-2
SCRIPT TO RUN ON EC2 (Give chmod 777 and chmod +x for the script) -
#!/bin/bash -x
sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version
res=$?
echo $res
mkdir /tmp/fabric-install/
cd /tmp/fabric-install/
wget https://aws-blockchain-templates-us-east-1.s3.us-east-1.amazonaws.com/hyperledger/fabric/templates/simplenetwork/latest/HyperLedger-BasicNetwork.tgz -O /home/ec2-user/HyperLedger-BasicNetwork.tgz
cd /home/ec2-user
tar xzvf HyperLedger-BasicNetwork.tgz
rm /home/ec2-user/HyperLedger-BasicNetwork.tgz
chown -R ec2-user:ec2-user HyperLedger-BasicNetwork
chmod +x /home/ec2-user/HyperLedger-BasicNetwork/artifacts/first-run-standalone.sh
/home/ec2-user/HyperLedger-BasicNetwork/artifacts/first-run-standalone.sh us-east-1 example.com org1 org2 org3 mychannel 354658284331.dkr.ecr.us-east-1.amazonaws.com/ 354658284331
res=$?
echo $res
IAM policy which I attached to the role -
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
]
}
NOTE -
Please replace the appropriate AWS ECR account number for your region and appropriate AWS region in the above script and script has (example.com org1 org2 org3 mychannel), Please change this too as per requirement. Its the same RootDomain, Org1SubDomain, Org2SubDomain, Org3SubDomain, ChannelName as we enter in the CF template).
This whole process is tested in the us-east-1 region. The script can be straight deployed in the us-east-1 region. To access the Hyperledger web monitor interface (http://EC2-DNS OR EIP:8080)
You should be Checking your IAM Role and It fixed my issue.

How to make 'aws ecr get-login' across regions?

I have a docker registry in AWS ECR in region 'us-east-1'. Everything works fine on EC2 instances launched in 'us-east-1'. But when I launch an instance in 'eu-central-1' and try to run
$(aws ecr get-login --region us-east-1)
I get the following response
Error response from daemon: Get https://acc-id.dkr.ecr.us-east-1.amazonaws.com/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
If I run
aws ecr get-login --region us-east-1
I do see the following response
docker login -u AWS -p xxxx -e none https://acc_id.dkr.ecr.us-east-1.amazonaws.com
Ec2 instance has the following policy for the iam-role:
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
"Resource": "*"
}
Please tell me how can I have this cross-region ECR accessibility.
Below procedure can be used for cross region image pull from ECR:
$(aws ecr get-login --no-include-email --region <region having repository> --registry-ids <id>)
docker pull <id>.dkr.ecr.us-west-1.amazonaws.com/<image_name>:<tag>
Below sample shows instance in region us-east-1 is pulling AWS Deep Learning Container Image from ECR hosted in us-west-1
To get region from EC2 instance metadata:
curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | grep region
"region" : "us-east-1"
ECR Login:
$(aws ecr get-login --no-include-email --region us-west-1 --registry-ids 763104351884)
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Docker Pull:
docker pull 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04
1.13-cpu-py27-ubuntu16.04: Pulling from tensorflow-training
34667c7e4631: Already exists
d18d76a881a4: Already exists
119c7358fbfc: Already exists
2aaf13f3eff0: Already exists
7b890657bd19: Already exists
f095a52e6583: Already exists
182b3abfb706: Already exists
89cc0ffab23c: Already exists
7a24716f4857: Already exists
Digest: sha256:65c7f58ac49ed39d5b6bde4f3800dbbf8d9f99b5316292b27315fb6a4b8be56b
Status: Downloaded newer image for 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04