AWS EKS "is not authorized to perform: iam:CreateServiceLinkedRole" - amazon-web-services

I have followed the documentation for spinning up an EKS cluster that says to make a service role with certain policies.
https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf
To create your Amazon EKS service role
1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. Choose Roles, then Create role.
3. Choose EKS from the list of services, then Allows Amazon EKS to manage your clusters on your behalf for your use case, then Next: Permissions.
4. Choose Next: Review.
5. For Role name, enter a unique name for your role, such as eksServiceRole, then choose Create role.
When I create a basic hello world app, it throws an AccessDenied error.
Error creating load balancer (will retry): failed to ensure load balancer for service default/nginx:
AccessDenied: User: arn:aws:sts::*************:assumed-role/eks-service-role/************* is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::*************:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
The two Policies that were added (AmazonEKSClusterPolicy, AmazonEKSServicePolicy) do not have the iam:CreateServiceLinkedRole action allowed. Are we supposed to add this outside of the policies defined in the guide? Or is this something that should be included in the EKS policies?

It seems that the EKS userguide assumes you have created load balancers in your AWS account prior to creating the EKS cluster, and thus have an existing AWSServiceRoleForElasticLoadBalancing service role in AWS IAM.
As described in https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/elb-service-linked-roles.html#create-service-linked-role
You don't need to manually create the AWSServiceRoleForElasticLoadBalancing role. Elastic Load Balancing creates this role for you when you create a load balancer.
EKS is attempting to do this for you, resulting in the access denied exception using the default policies.
Other options to explicitly create the service-linked role prior to EKS cluster creation include:
AWS CLI
aws iam create-service-linked-role --aws-service-name "elasticloadbalancing.amazonaws.com"
Terraform
resource "aws_iam_service_linked_role" "elasticloadbalancing" {
aws_service_name = "elasticloadbalancing.amazonaws.com"
}
Or, manually create a load balancer from the UI Console.
Regardless of provisioning options, you should know things will work when you see the following role in AWS IAM
arn:aws:iam::<ACCOUNT_ID>:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing

I got it worked by adding this policy to the EKS Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAccountAttributes"
],
"Resource": "*"
}
]
}

Related

AWS Allow Cross-account EKS Cluster to Pull Images from ECR

Summary:
I'm looking to enable EKS nodes to pull images from an ECR registry from a different AWS project. I created an "AllowPull" policy in the desired ECR repository and set the principal of the policy to the ARN of the EKS cluster role, but node is unable to pull the image.
How should the policy be formulated in order to allow all nodes in an EKS cluster to pull from a cross-account ECR repository?
Attempt Details:
The ECR registry recourse name is:
arn:aws:ecr:us-east-2:226427918358:repository/external-pull-test
The EKS cluster that needs to pull the images has the following role attached:
arn:aws:iam::02182452XXXX:role/aws-dev-eks-cluster-crpiXXXX091410594876160000000c
The external ECR registry has the following policy JSON:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPull",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::02182452XXXX:role/aws-dev-eks-cluster-crpiXXXX091410594876160000000c"
},
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ecr:DescribeRepositories",
"ecr:GetDownloadUrlForLayer"
]
}
]
}
The pod ImagePullBackOff error specifies that the user attempting to authenticate with the registry is this assumed role:
arn:aws:sts::02182452XXXX:assumed-role/aws-dev-eks-cluster-crpiXXXX091410594876160000000c/i-0ea4f53b6dfdcxxxx
Environment:
Kubernetes: v1.16.15-eks-e1a842
Additional Details:
Using the ARN of my user principal (cross-account) in the policy did allow me to pull images using docker locally. Using the ARN of the assumed role did enable the node to pull the image, but my understanding is that configuring the policy with a particular assumed role won't guarentee that the cluster nodes can consistently pull from the registry.
Another method is click on the "external-pull-test" repo on the ECR console, on the left panel under "Repositories" click on "Permissions", then click on "Edit" on the top right. You can add the account ID that needs to pull from this repo at "AWS account IDs". Check the permitted actions at the bottom "Actions" drop down box. "Save" and you should be able to pull.

Kubernetes Route53 external-dns issue

I am running a K8S cluster in AWS with EKS and external-DNS with an ingress-Nginx-ingress-controller. Application is working fine and accessible on load-balancer. Now I am trying to point my DNS to this cluster i.e. e1.exapmle.com and for this, I am following below documentation.
https://github.com/kubernetes-sigs/external-dns
The issue I am facing is that I have two AWS accounts (QA, Staging) and hosted zones are created in staging account and the cluster is running on QA account. I have tried cross-account IAM roles for this to work but I can only see hosted zones via the web console, cli or external-DNS pods are not able to able to communicate with Route53.
This is what I have done so far:
Create an IAM policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets"
],
"Resource": [
"arn:aws:route53:::hostedzone/*"
]
},
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones",
"route53:ListResourceRecordSets"
],
"Resource": [
"*"
]
}
]
}
Create a cross-account IAM role trusting account staging and trusted account QA.
Assum policy in the trusted account by specifying IAM role ARN for the above policy.
By doing above I only managed to see hosted zones via web console when I switch roles.
I assume you provided a role arn of a role in the EKS account to the extern-dns pods which is allowed to assume to role inside you HostedZone account!?
This will not work because then (as you do inside the console) you need to explicitly assume the role with existing permissions.
You need to pass the Cross-Account role with your attached route53 policy directly to the external-dns pods.
Pod -> assumes role in different account -> receives temp. credentials
pod -> assumed role in eks account (-> allowd to assume cross-accout role)
last step won’t happen i guess

AWS IAM Role permission issue

We have just built a new Things Enterprise server hosted at AWS on an EC2 instance and created an application to use AWS IOT. We are getting the following error
“message”: “User: arn:aws:sts::446971925991:assumed-role/Things-Enterprise-Stack-Srv-StackIAMRole-DBHBSMSY05AQ/i-095895d605fab3fa4 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::446971925991:role/Bosh-Parking-IOT-Stack-TheThingsStackRoleCD9FBAD2-C44RRJJ53M93”
I have been told
What is the execution role of the TTES instance that is trying to assume the role? The role TTES needs to be able to assume that role. That will give the right permissions.
But I'm not sure what that means, i'm presuming i need to add / alter some permissions within an IAM role. Can someone point me in the right direction Pls.
From the error message it seems that your IAM role for Amazon EC2 has no permissions to assume a role Bosh-Parking-IOT-Stack-TheThingsStackRoleCD9FBAD2-C44RRJJ53M93.
To add such permissions manually you can do the following:
Go to IAM Console->Roles.
In the Roles window, you can use Search bar to locate Things-Enterprise-Stack-Srv-StackIAMRole-DBHBSMSY05AQ role.
Once you find the role, you click on Add inline policy.
Once Create policy window shows, you can go to JSON tab and add the following JSON policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAssumeRole",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::446971925991:role/Bosh-Parking-IOT-Stack-TheThingsStackRoleCD9FBAD2-C44RRJJ53M93"
}
]
}
Then click Review Policy, name the policy (e.g. PolicyToAssumeRole) and Create policy
However, based on your policy names (e.g. Stack-Srv-StackIAMRole) it is possible that they have been create by CloudFormation. If this is the case, then manually changing the roles as described above is a bad practice and will lead to drift. Any changes to resources created by CloudFormation should be done using CloudFormation. Sadly, your question does not provide any details about CloudFormation templates used, therefore its difficult to comment on that more.

AWS ECS Service Definition: Role property

I am setting up an AWS ECS Service using cloudformation and yaml syntax.
At some point, in the relevant documentation there is a property called Role whose definitions is the following:
Role
The name or ARN of an AWS Identity and Access Management (IAM) role that allows your Amazon ECS container agent to make calls to your
load balancer.
Note In some cases, you might need to add a dependency on the service role's policy. For more information, see IAM role policy in
DependsOn Attribute.
Required: No
Type: String
Update requires: Replacement
Since I intend to place the specific service behind an Application Load Balancer, is this property needed?
If so, do I need to create a new policy or are there any pre-defined policies that can serve this purpose?
Would it be enough if I just added the role/policy on the EC2 container instance level (e.g. append it in the relevant template that creates the ECS cluster offered by amazon?)
I would really appreciate any examples or use cases if any because the documentation is vague and incomplete on the topic.
You need the Role attribute if you want to use the Application Load balancer with your ECS service. As per the description the role allows your ECS service agent to connect to load balancer. If you are not using load balancer then, the field is optional.
Also setting the role on EC2 instance level is not needed. Since there are some policies which are related to ECS services, adding it at instance level role is not valid.
Please find below AWS managed polices to create the role:
{
"AttachedPolicies": [
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole",
"PolicyName": "AmazonEC2ContainerServiceAutoscaleRole"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
"PolicyName": "AmazonEC2ContainerServiceforEC2Role"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole",
"PolicyName": "AmazonEC2ContainerServiceRole"
}
]
}
Trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
To register your service with application load balancer you need the role for the service itself, but some permission needs for ECS container to register with cluster.
Amazon Elastic Container Service uses AWS Identity and Access
Management (IAM) service-linked roles. A service-linked role is a
unique type of IAM role that is linked directly to Amazon ECS.
Service-linked roles are predefined by Amazon ECS and include all the
permissions that the service requires to call other AWS services on
your behalf.
A service-linked role makes setting up Amazon ECS easier because you
don’t have to manually add the necessary permissions. Amazon ECS
defines the permissions of its service-linked roles, and unless
defined otherwise, only Amazon ECS can assume its roles. The defined
permissions include the trust policy and the permissions policy, and
that permissions policy cannot be attached to any other IAM entity.
So if you check this role this contain property
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
To Debug remove the role to service , it will not register.
If you create service using console by default it attach the role.
Amazon ECS needs permissions to register and deregister container
instances with your load balancer when tasks are created and stopped.
In most cases, the Amazon ECS service role is automatically created
for you in the Amazon ECS console first run experience. You can use
the following procedure to check and see if your account already has
an Amazon ECS service role.
This managed polici by AWS having ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Policy ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Description Default policy for Amazon ECS service role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
}
]
}
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/check-service-role.html
this is how its look like
If you do not assign a role to ECS container instance it will never show in you ECS cluster and it will throw an error
2018-09-06T15:26:22Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
under /var/logs/ecs
tail -f ecs-agent.log.2018-09-06
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html

Adding permission to IAM role

I want to add permissions to my instance's IAM roles to allow for the use of Cloudwatch Log Service. Upon looking at the Configuration Guide for the service, I see the following passage:
The CloudWatch Logs agent supports IAM roles and users.
If your instance already has an IAM role associated with it, make sure that you include the IAM policy below.
If you don't already have an IAM role assigned to your instance,
you'll need to use your IAM credentials for the next steps because you cannot assign an IAM role to an existing instance;
you can only specify a role when you launch a new instance.
I'm having a hard time figuring exactly what this means. When I created a new instance, I made an IAM role with the CloudWatchLogsFullAccess policy and an inline policy the configuration guide tells you to add:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"
],
"Resource": [
"arn:aws:logs:*:*:*"
]
}
]
}
This was for a new instance, however the above passage confuses me about adding permissions to existing instances. Can I just add the CloudWatchLogsFullAccess policy to the existing instance's role and that inline policy as well?
While it was true an IAM role can be assigned to an instance only during instance launch, AWS announced in February 2017 that it is now possible to replace or attach an IAM role to an existing instance.
Using AWS Console: Easily Replace or Attach an IAM Role to an Existing EC2 Instance by Using the EC2 Console
Using AWS CLI: Attach an AWS IAM Role to an Existing Amazon EC2 Instance by Using the AWS CLI