Summary:
I'm looking to enable EKS nodes to pull images from an ECR registry from a different AWS project. I created an "AllowPull" policy in the desired ECR repository and set the principal of the policy to the ARN of the EKS cluster role, but node is unable to pull the image.
How should the policy be formulated in order to allow all nodes in an EKS cluster to pull from a cross-account ECR repository?
Attempt Details:
The ECR registry recourse name is:
arn:aws:ecr:us-east-2:226427918358:repository/external-pull-test
The EKS cluster that needs to pull the images has the following role attached:
arn:aws:iam::02182452XXXX:role/aws-dev-eks-cluster-crpiXXXX091410594876160000000c
The external ECR registry has the following policy JSON:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::02182452XXXX:role/aws-dev-eks-cluster-crpiXXXX091410594876160000000c"
},
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ecr:DescribeRepositories",
"ecr:GetDownloadUrlForLayer"
]
}
]
}
The pod ImagePullBackOff error specifies that the user attempting to authenticate with the registry is this assumed role:
arn:aws:sts::02182452XXXX:assumed-role/aws-dev-eks-cluster-crpiXXXX091410594876160000000c/i-0ea4f53b6dfdcxxxx
Environment:
Kubernetes: v1.16.15-eks-e1a842
Additional Details:
Using the ARN of my user principal (cross-account) in the policy did allow me to pull images using docker locally. Using the ARN of the assumed role did enable the node to pull the image, but my understanding is that configuring the policy with a particular assumed role won't guarentee that the cluster nodes can consistently pull from the registry.
Another method is click on the "external-pull-test" repo on the ECR console, on the left panel under "Repositories" click on "Permissions", then click on "Edit" on the top right. You can add the account ID that needs to pull from this repo at "AWS account IDs". Check the permitted actions at the bottom "Actions" drop down box. "Save" and you should be able to pull.
Related
I have an EKS cluster and EC2. I would like to create an instance profile and attach it to the EC2 - this profile should allow ONLY READ access to the EKS cluster.
Will the following policy be apt for this requirement?:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"eks:ListNodegroups",
"eks:DescribeFargateProfile",
"eks:ListTagsForResource",
"eks:ListAddons",
"eks:DescribeAddon",
"eks:ListFargateProfiles",
"eks:DescribeNodegroup",
"eks:ListUpdates",
"eks:DescribeUpdate",
"eks:AccessKubernetesApi",
"eks:DescribeCluster",
"eks:ListClusters",
"eks:DescribeAddonVersions"
],
"Resource": "*"
}
]
}
Depends on what you mean by "read" access.
If you mean "read" from AWS perspective as in being able to use the AWS CLI to tell you about EKS, yes that would be sufficient to get you started. This will not include any kubectl commands.
But if you mean read as in being able to execute kubectl commands against the cluster, then you will not be able to achieve that with this.
To implement read access to the cluster itself using kubectl commands, your EC2 instance will need a minimum of the following IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster",
"eks:ListClusters"
],
"Resource": "*"
}
]
}
With this, your EC2 will be able to execute eksctl utils write-kubeconfig --cluster=cluster-name to configure the kubeconfig. This also assumes you have the required components installed on your EC2 to run kubectl.
You also need to set up permissions within your cluster because the IAM permissions alone don't actually grant any visibility within the cluster itself.
The role you assign to your EC2 would need to be added to the aws-auth configmap in the kube-system namespace. See Managing users or IAM roles for your cluster from AWS docs for examples.
Unfortunately I don't believe there's a simple out-of-the-box RBAC role you can use that gives you read-only access to the entire cluster. Kubernetes provides four user-facing roles and of them, only the system:masters group has cluster-wide access.
Have a look at Kubernetes Using RBAC Authorization documentation - specifically on user-facing roles.
You will need to design a permission strategy to fit your needs, but you do have the default role view that you can start from. The default view user-facing role is tied to a ClusterRoleBinding and was designed / intended to be used in a namespace specific capacity.
Permissions and RBAC for Kubernetes is a very deep rabbit-hole.
I am setting up an AWS ECS Service using cloudformation and yaml syntax.
At some point, in the relevant documentation there is a property called Role whose definitions is the following:
Role
The name or ARN of an AWS Identity and Access Management (IAM) role that allows your Amazon ECS container agent to make calls to your
load balancer.
Note In some cases, you might need to add a dependency on the service role's policy. For more information, see IAM role policy in
DependsOn Attribute.
Required: No
Type: String
Update requires: Replacement
Since I intend to place the specific service behind an Application Load Balancer, is this property needed?
If so, do I need to create a new policy or are there any pre-defined policies that can serve this purpose?
Would it be enough if I just added the role/policy on the EC2 container instance level (e.g. append it in the relevant template that creates the ECS cluster offered by amazon?)
I would really appreciate any examples or use cases if any because the documentation is vague and incomplete on the topic.
You need the Role attribute if you want to use the Application Load balancer with your ECS service. As per the description the role allows your ECS service agent to connect to load balancer. If you are not using load balancer then, the field is optional.
Also setting the role on EC2 instance level is not needed. Since there are some policies which are related to ECS services, adding it at instance level role is not valid.
Please find below AWS managed polices to create the role:
{
"AttachedPolicies": [
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole",
"PolicyName": "AmazonEC2ContainerServiceAutoscaleRole"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
"PolicyName": "AmazonEC2ContainerServiceforEC2Role"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole",
"PolicyName": "AmazonEC2ContainerServiceRole"
}
]
}
Trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
To register your service with application load balancer you need the role for the service itself, but some permission needs for ECS container to register with cluster.
Amazon Elastic Container Service uses AWS Identity and Access
Management (IAM) service-linked roles. A service-linked role is a
unique type of IAM role that is linked directly to Amazon ECS.
Service-linked roles are predefined by Amazon ECS and include all the
permissions that the service requires to call other AWS services on
your behalf.
A service-linked role makes setting up Amazon ECS easier because you
don’t have to manually add the necessary permissions. Amazon ECS
defines the permissions of its service-linked roles, and unless
defined otherwise, only Amazon ECS can assume its roles. The defined
permissions include the trust policy and the permissions policy, and
that permissions policy cannot be attached to any other IAM entity.
So if you check this role this contain property
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
To Debug remove the role to service , it will not register.
If you create service using console by default it attach the role.
Amazon ECS needs permissions to register and deregister container
instances with your load balancer when tasks are created and stopped.
In most cases, the Amazon ECS service role is automatically created
for you in the Amazon ECS console first run experience. You can use
the following procedure to check and see if your account already has
an Amazon ECS service role.
This managed polici by AWS having ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Policy ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Description Default policy for Amazon ECS service role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
}
]
}
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/check-service-role.html
this is how its look like
If you do not assign a role to ECS container instance it will never show in you ECS cluster and it will throw an error
2018-09-06T15:26:22Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
under /var/logs/ecs
tail -f ecs-agent.log.2018-09-06
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html
I have followed the documentation for spinning up an EKS cluster that says to make a service role with certain policies.
https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf
To create your Amazon EKS service role
1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. Choose Roles, then Create role.
3. Choose EKS from the list of services, then Allows Amazon EKS to manage your clusters on your behalf for your use case, then Next: Permissions.
4. Choose Next: Review.
5. For Role name, enter a unique name for your role, such as eksServiceRole, then choose Create role.
When I create a basic hello world app, it throws an AccessDenied error.
Error creating load balancer (will retry): failed to ensure load balancer for service default/nginx:
AccessDenied: User: arn:aws:sts::*************:assumed-role/eks-service-role/************* is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::*************:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
The two Policies that were added (AmazonEKSClusterPolicy, AmazonEKSServicePolicy) do not have the iam:CreateServiceLinkedRole action allowed. Are we supposed to add this outside of the policies defined in the guide? Or is this something that should be included in the EKS policies?
It seems that the EKS userguide assumes you have created load balancers in your AWS account prior to creating the EKS cluster, and thus have an existing AWSServiceRoleForElasticLoadBalancing service role in AWS IAM.
As described in https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/elb-service-linked-roles.html#create-service-linked-role
You don't need to manually create the AWSServiceRoleForElasticLoadBalancing role. Elastic Load Balancing creates this role for you when you create a load balancer.
EKS is attempting to do this for you, resulting in the access denied exception using the default policies.
Other options to explicitly create the service-linked role prior to EKS cluster creation include:
AWS CLI
aws iam create-service-linked-role --aws-service-name "elasticloadbalancing.amazonaws.com"
Terraform
resource "aws_iam_service_linked_role" "elasticloadbalancing" {
aws_service_name = "elasticloadbalancing.amazonaws.com"
}
Or, manually create a load balancer from the UI Console.
Regardless of provisioning options, you should know things will work when you see the following role in AWS IAM
arn:aws:iam::<ACCOUNT_ID>:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
I got it worked by adding this policy to the EKS Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAccountAttributes"
],
"Resource": "*"
}
]
}
I have an Amazon account with a K8S cluster which is able to pull images from the same account's ECR repository.
But, my company have another account with another ECR repository. How can I pull image from this "external" ECR repository ?
I'am also a Rancher user and I used to do this by installing a special container (https://github.com/rancher/rancher-ecr-credentials) which does the job.
Is there something equivalent for Kubernetes?
Thanks for your precious help
Since you already have this setup for pulling images from the same account, you can do this with IAM policy level or ECR permissions, in your other AWS account set up a policy specifying the AWS account number (where k8s is) that will be able to pull images
For example grant pull permissions in the ECR Permissions tab
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "k8s-aws-permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::aws_account_number:root"
},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
]
}
]
}
I have a custom docker image uploaded to ECS. I opened up the permissions to try and get through this issue (I will lock it down again once I can get this to work). I am attempting to deploy the docker image to elastic beanstalk. I have a docker enabled elastic beanstalk environment set up. According to the AWS docs, if I am pulling my image from within AWS, I don't need to pass in credentials. So I upload my Dockerrun.aws.json file and attempt to install it. It fails with the error:
Command failed on instance. Return code: 1 Output: Failed to authenticate with ECR for registry '434875166128' in 'us-east-1'. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03build.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
The /var/log/eb-activity.log information has nothing useful in it.
Here's my Dockerrun.aws.json file:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "{id000xxxx}.dkr.ecr.us-east-1.amazonaws.com/my-repo:1.0.0",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "4000"
}
],
"Logging": "/var/log/app-name"
}
I have also tried adding the authentication with the dockercfg.json file in S3. It didn't work for me either.
Note that I am using a business account instead of a personal account, so there may be some unknown variances as well.
Thanks!
Update: My user has full permissions at the moment too, so there shouldn't be anything permission-wise getting in the way.
I was having the same problem.
Solution:
In AWS -> IAM -> Roles - > pick the role your beanstalk is using.
In my case it was set to aws-elasticbeanstalk-ec2-role
Under Permissions for the role, attach policy: AmazonEC2ContainerRegistryReadOnly
In ECR there is no need to give any permissions to this role.
Assuming
You are using Terraform to provision your infrastructure
You have created a sample ElasticBeanstalk app at least once, so that you have the default role created.
The default ElasticBeanstalk role is named: aws-elasticbeanstalk-ec2-role
Then you can comfortably use the following format to add ECR Read Only policy to the role:
data "aws_iam_role" "elastic_beanstalk_role" {
name = "aws-elasticbeanstalk-ec2-role"
}
resource "aws_iam_policy" "ebs_ecr_policy" {
name = "aws-elasticbeanstalk-ec2-ecr-policy"
description = "Enable elastic-beanstalk to be able to access ECR repository with images"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:GetLifecyclePolicy",
"ecr:GetLifecyclePolicyPreview",
"ecr:ListTagsForResource",
"ecr:DescribeImageScanFindings"
],
"Resource": "*"
}
]
}
EOF
}
resource "aws_iam_policy_attachment" "ebs_ecr-policy-attach" {
name = "ebs-ecr-policy-attachment"
roles = [data.aws_iam_role.elastic_beanstalk_role.name]
policy_arn = aws_iam_policy.ebs_ecr_policy.arn
}
This way you can manage updates to the role and policy from your infrastructure code.
You can intialize necessary service roles for elastic beanstalk (aws-elasticbeanstalk-ec2-role , aws-elasticbeanstalk-service-role , AWSServiceRoleForECS ) by using the new console of Elastic Beanstalk.
You have to do this only one time on each AWS account :
Go to the Elastic beanstalk console.
Accept the "new design" : in the top of the console, if see a message "we re testing a new design", optin to accept to use the new version of the console. Warning, it seems you cant rollback to the old console.
Start the Create New Application wizard, and use a default sample application in the technology.
Complete all the step of the wizard until the resume, and look at the Security pannel : you will see the two roles "aws-elasticbeanstalk-ec2-role" and "aws-elasticbeanstalk-service-role". And terminate the wizard to create the sample app.
After a while, the application should be running
In case of emergency, go to the IAM console and delete the roles aws-elasticbeanstalk-ec2-role and aws-elasticbeanstalk-service-role and run the wizard again.
I fixed the "Command failed on instance. Return code: 1 Output: Failed to authenticate with ECR for registry" and an other strange error ("The AWS Access Key Id you provided does not exist in our records. (ElasticBeanstalk::ManifestDownloadError)") by using the NEW console. I still had this error with the old one.