I'm new to using AWS EBS and ECS, so please forgive me if I ask a question which might be obvious for others. To the issue:
I've configured EBS and ECS in the same AWS cloud region. I've pushed my container and can see the image in ECS. The EBS environment has been built using a Dockerrun.aws.json pointing to the ECS. During the build of the EBS I get the following error:
2020/11/02 20:50:25.858567 [INFO] authenticate with ECR if the image is in an ECR repo
2020/11/02 20:50:25.858582 [INFO] Running command /bin/sh -c aws ecr get-login --no-include-email --registry-ids 731178912345 --region eu-west-1
2020/11/02 20:50:28.355911 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to authenticate with ECR for registry 731178912XXX in eu-west-1: Command /bin/sh -c aws ecr get-login --no-include-email --registry-ids 731178912XXX --region eu-west-1 failed with error exit status 255. Stderr:
An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:sts::731178912XXX:assumed-role/aws-elasticbeanstalk-ec2-role/i-0bff9b5324348ea71 is not authorized to perform: ecr:GetAuthorizationToken on resource: *
I read somewhere that the identify is automatically passed in to process the authentication. I couldn't find any indication on what to do to resolve the issue. Maybe someone could help me out?
After adding the permission as discussed I get these:
2020/11/03 09:17:46.490399 [INFO] pull docker image if update is not false in Dockerrun.aws.json
2020/11/03 09:17:46.490431 [INFO] Running command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest
2020/11/03 09:17:46.576648 [WARN] failed to execute command: docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest, retrying...
2020/11/03 09:17:46.576673 [INFO] Running command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest
2020/11/03 09:17:46.659379 [ERROR] An error occurred during execution of command [app-deploy] - [Docker Specific Build Application]. Stop running the command. Error: failed to pull docker image: Command /bin/sh -c docker pull 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project:latest failed with error exit status 1. Stderr:Error response from daemon: pull access denied for 731178912XXX.dkr.ecr.eu-west-1.amazonaws.com/user/project, repository does not exist or may require 'docker login': denied: User: arn:aws:sts::731178912XXX:assumed-role/aws-elasticbeanstalk-ec2-role/i-0aa453e0d44cdfa90 is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:eu-west-1:731178912XXX:repository/user/project
Your aws-elasticbeanstalk-ec2-role instance role does not have permissions to execute ecr:GetAuthorizationToken.
You can address this, by adding an inline policy to the aws-elasticbeanstalk-ec2-role role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
}
]
}
Please note that more permissions may be needed after this one. You can add them in the same way.
Related
I am trying to push docker image to ECR from my mac. I added identity to aws-vault, but using aws-vault failed to obtain password. My peer has the same privilege and he can login to ECR without problem.
.aws % aws-vault list
Profile Credentials Sessions
======= =========== ========
identity identity -
.aws % ENCRYPTED_PASSWORD=$(aws-vault exec identity -- aws ecr get-login-password --region us-west-2)
An error occurred (AccessDeniedException) when calling the GetAuthorizationToken operation: User: arn:aws:iam::97xxxxxxxxxx:user/xxxxx.xxxxxx#xxxxxxxxx.com is not authorized to perform: ecr:GetAuthorizationToken on resource: * with an explicit deny in an identity-based policy
We are using the same role. I intend to execute the following line once ENCRYPTED_PASSWORD is obtained--
aws-vault exec identity -- aws ecr --region us-west-2 | docker login -u AWS -p ${ENCRYPTED_PASSWORD} xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/my-ecr
My .aws/config has this line refer to the above.
[profile identity]
Since my teammate has the same policy/permission, I don't see "GetAuthorizationToken" as an issue. What else? I'm trying to find a way to push my docker images from Mac to the ECR.
Completely redo the config, linked all roles to identity.
[profile xx-xxxx-services-monitoring]
role_arn=arn:aws:iam::xxxxxxxxxxxx:role/XXMonitoring
source_profile=identity
sparent_profile=mfa
[profile identity]
Now is the token issue-- I entered the code from MFA Authenticator but failed
.aws % aws-vault exec xx-xxxx-main-admin -- aws ecr get-login --region us-west-2
Warning: parent_profile is deprecated, please use include_profile instead in your AWS config
Enter token for arn:aws:iam::xxxxxxxxxxxx:mfa/xxxxxx.xxxx#xxxxxx.com: **125492**
aws-vault: error: exec: Failed to get credentials for xx-xxxx-main-admin: AccessDenied: MultiFactorAuthentication failed, unable to validate MFA code. Please verify your MFA serial number is valid and associated with this user.
status code: 403, request id: 25808115-d9fa-480d-87d4-xxxxxxxxxxxxx
Doesn't look like the token is valid from Authenticator MFA
first aws-vault "GetAuthorizationToken" was caused by an unhealthy, ilformatted ~/.aws/config file. Roles must be specified with the following format--
[profile xx-xxxx-services-monitoring]
role_arn=arn:aws:iam::xxxxxxxxxxxx:role/XXMonitoring
source_profile=identity
sparent_profile=mfa
[profile identity]
Second part is MFA issue. MFA serial number must match what you configured in your authenticator. (DUO, Authy, Authenticator...) Your ~/.aws/config should have this section--
[profile mfa]
mfa_serial=arn:aws:iam::xxxxxxxxxxxx:mfa/xxxxx.xxxx#xxxxx.com
COMMAND_EXECUTION_ERROR: Error while executing command: $(aws ecr get-login --no-include-email --region us-east-1). Reason: exit status 127
Below is my buildspec.yml file
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- $(aws ecr get-login --region ***-east-*)
- REPOSITORY_URI=***********.dkr.ecr.***-east-*.amazonaws.com/repositoryname
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION)
- IMAGE_TAG=${COMMIT_HASH:=latest}
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
- echo Writing definitions file...
- printf '[{"name":"project-container","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > taskdefinition.json
artifacts:
files: taskdefinition.json
In case it helps someone else, for the work I'm doing inside my build scripts executed by CodeBuild. These are the IAM permissions I had to add (finding them one by one as I hit the error).
{
"Action": [
"ecr:GetAuthorizationToken",
"ecr:DescribeRepositories",
"ecr:CreateRepository",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:BatchCheckLayerAvailability",
"ecr:PutImage",
"ecs:UpdateService"
],
"Resource": "*",
"Effect": "Allow"
} '
I'm sure there are more permissions that may be required if you're doing stuff I'm not doing in your builds. I'm pushing to ECR and forcing the Service (and the related tasks) to deploy the new image.
Your post has inconsistent details, is that intentional? If not, it may be causing the problem. Your code snippet says:
$(aws ecr get-login --region ***-east-*)
Perhaps you purposely redacted the region (what's the point of that btw?) but why is it missing the --no-include-email? Higher up in your post, you do make reference to --no-include-email, so I know you're aware of it.
Run the process outside of a subshell to get a better log
Rather than running it inside a subshell (e.g. $(my command)), for troubleshooting purposes, try running taking the subshell out so you can get better output. Report the results here so we can troubleshoot the error you get.
aws ecr get-login --no-include-email --region us-east-1 <- try this temporarily
vs.
$(aws ecr get-login --no-include-email --region us-east-1)
Have you created an IAM Policy with ECR permissions for CodeBuild to use?
This is very important. CodeBuild needs permission to access ECR on your behalf. Here's an example I found on this blog article. It may need tweaking to your needs. http://beta.awsdocs.com/services/code_build/build_docker_images/
{
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:GetAuthorizationToken",
"ecr:InitiateLayerUpload",
"ecr:PutImage",
"ecr:UploadLayerPart"
],
"Resource": "*",
"Effect": "Allow"
}
I'm having trouble setting up Spinnaker with ECR access.
Background: I installed spinnaker using helm on an EKS cluster and I've confirmed that the cluster has the necessary ECR permissions (by manually running ECR commands from within the clouddriver pod). I am following the instructions here to get Spinnaker+ECR set up: https://www.spinnaker.io/setup/install/providers/docker-registry/
Issue: When I run:
hal config provider docker-registry account add my-ecr-registry \
--address $ADDRESS \
--username AWS \
--password-command "aws --region us-west-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | sed 's/^AWS://'"
I get the following output:
+ Get current deployment
Success
- Add the some-ecr-registry account
Failure
Problems in default.provider.dockerRegistry.some-ecr-registry:
- WARNING Resolved Password was empty, missing dependencies for
running password command?
- WARNING You have a supplied a username but no password.
! ERROR Unable to fetch tags from the docker repository: code, 400
Bad Request
? Can the provided user access this repository?
- WARNING None of your supplied repositories contain any tags.
Spinnaker will not be able to deploy any docker images.
? Push some images to your registry.
Problems in halconfig:
- WARNING There is a newer version of Halyard available (1.28.0),
please update when possible
? Run 'sudo apt-get update && sudo apt-get install
spinnaker-halyard -y' to upgrade
- Failed to add account some-ecr-registry for provider
dockerRegistry.
I have confirmed that the aws-cli is installed on the clouddriver pod. And I've confirmed that I can the password-command directly from the clouddriver pod and it successfully returns a token.
I've also confirmed that if I manually generate an ECR token and run hal config provider docker-registry account add my-ecr-registry --address $ADDRESS --username AWS --password-command "echo $MANUALLY_GENERATED_TOKEN" everything works fine. So there is something specific to the password-command that is going wrong and I'm not sure how to debug this.
One other odd behavior: if I simplify the password command to be: hal config provider docker-registry account add some-ecr-registry --address $ADDRESS --username AWS --repositories code --password-command "aws --region us-west-2 ecr get-authorization-token" , I get an addt'l piece of output that says "- WARNING Password command returned non 0 return code stderr/stdout was:bash: aws: command not found". This output only appears for this simplified command.
Any advice on how to debug this would be much appreciated.
If like me your ECR registry is in another account, then you have to forcibly assume the role for the target account where your registry resides
passwordCommand: read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< `aws sts assume-role --role-arn arn:aws:iam::<AWS_ACCOUNT>:role/<SPINNAKER ROLE_NAME> --query "[Credentials.AccessKeyId, Credentials.SecretAccessKey, Credentials.SessionToken]" --output text --role-session-name spinnakerManaged-w2`; export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN; aws ecr get-authorization-token --region us-west-2 --output text --query 'authorizationData[].authorizationToken' --registry-ids <AWS_ACCOUNT> | base64 -d | sed 's/^AWS://'
Credits to https://github.com/spinnaker/spinnaker/issues/5374#issuecomment-607468678
I also installed Spinnaker on AKS and all i did was by using an AWS Managing User with the correct AWS IAM policy to ECR:* i have access to the ECR repositories directly.
I dont think that hal being java based will execute the Bash command in --password-command
set the AWS ECS provider in your spinnaker deployment
Use the Following AWS IAM policy (SpinnakerManagingPolicy) to be attached to the AWS MAnaging User to give access to ECR. Please replace the AWS Accounts based on your need.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:*",
"cloudformation:*",
"ecr:*"
],
"Resource": [
"*"
]
},
{
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::123456789012:role/SpinnakerManagedRoleAccount1",
"arn:aws:iam::101121314157:role/SpinnakerManagedRoleAccount2",
"arn:aws:iam::202122232425:role/SpinnakerManagedRoleAccount3"
],
"Effect": "Allow"
}
]
}
Iam configuring CICD piepline usng aws services code build pilepileline etc, to update the deploymnet in my ecs fargate cluster, In my buildspec.yml file aws ecs cli commands are failing throwing "COMMAND_EXECUTION_ERROR: Error while executing command: aws ecs update-service --services xxxxx. Reason: exit status 255"
I have tried providing permission to the codebuild role with "AmazonECS_FullAccess" policy.
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
- echo Writing image definitions file...
- printf '[{"name":"xxxxxxx","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
- cat imagedefinitions.json
- echo Updating ECSfargate service ...
- aws ecs describe-services --services xxxxxxxxxxxxxxxxx
- aws ecs update-service --service xxxxxxxxxxxxxxxxx --desired-count 2 --no-force-new-deployment
Attaching a policy of AmazonEC2ContainerRegistryPowerUser fixed this issue for me.
Error Code 255 means:
255 -- Command failed. There were errors thrown by either the CLI or by the service the request was made to.
https://docs.aws.amazon.com/cli/latest/topic/return-codes.html
As you mentioned you have provided full ECS access to CB role, next thing you can check is why the command is failing: Is it failing on ecs describe or ecs update-service? Since if the Fargate Service did not stabilize, it will result in error 255.
I would suggest to:
1) Just leave the ecs describe command and see if that works.
2) If (1) is successful, then do the ecs update-service and monitor the service in AWS ECS console and/or CW logs group if you Fargate Taskdef has a logGroup entry.
You'll need to grant the permission GetAuthorizationToken of your pipeline to
ecr:*
instead of to
ecr:repository/<you_repo>
because
aws ecr get-login-password --region <aws_region>
is executed against
<account_id>.dkr.ecr.<aws_region>.amazonaws.com
and not against
<account_id>.dkr.ecr.<aws_region>.amazonaws.com/repository/<your_repo>
I have had the same problem as you and the way I fixed it was by following: try going to CodeBuild and then to its IAM Role. AmazonEC2ContainerRegistryFullAccess role and now click on 'Edit' for that code build and select 'Environment' and click on Allow AWS CodeBuild to modify this service role so it can be used with this building project. Now try again.
Using PrivilegedMode mode in the CodeBuild project. The mode is required when building a docker image inside a docker container.
Cheers
I have a docker registry in AWS ECR in region 'us-east-1'. Everything works fine on EC2 instances launched in 'us-east-1'. But when I launch an instance in 'eu-central-1' and try to run
$(aws ecr get-login --region us-east-1)
I get the following response
Error response from daemon: Get https://acc-id.dkr.ecr.us-east-1.amazonaws.com/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
If I run
aws ecr get-login --region us-east-1
I do see the following response
docker login -u AWS -p xxxx -e none https://acc_id.dkr.ecr.us-east-1.amazonaws.com
Ec2 instance has the following policy for the iam-role:
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:BatchGetImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload",
"ecr:PutImage"
],
"Resource": "*"
}
Please tell me how can I have this cross-region ECR accessibility.
Below procedure can be used for cross region image pull from ECR:
$(aws ecr get-login --no-include-email --region <region having repository> --registry-ids <id>)
docker pull <id>.dkr.ecr.us-west-1.amazonaws.com/<image_name>:<tag>
Below sample shows instance in region us-east-1 is pulling AWS Deep Learning Container Image from ECR hosted in us-west-1
To get region from EC2 instance metadata:
curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | grep region
"region" : "us-east-1"
ECR Login:
$(aws ecr get-login --no-include-email --region us-west-1 --registry-ids 763104351884)
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Docker Pull:
docker pull 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04
1.13-cpu-py27-ubuntu16.04: Pulling from tensorflow-training
34667c7e4631: Already exists
d18d76a881a4: Already exists
119c7358fbfc: Already exists
2aaf13f3eff0: Already exists
7b890657bd19: Already exists
f095a52e6583: Already exists
182b3abfb706: Already exists
89cc0ffab23c: Already exists
7a24716f4857: Already exists
Digest: sha256:65c7f58ac49ed39d5b6bde4f3800dbbf8d9f99b5316292b27315fb6a4b8be56b
Status: Downloaded newer image for 763104351884.dkr.ecr.us-west-1.amazonaws.com/tensorflow-training:1.13-cpu-py27-ubuntu16.04