Im trying to use EC2 Container service. Im using terraform for creating it.
I have defined a ecs cluster, autoscaling group, launch configuration. All seems to work. Except one thing. The ec2 instances are creating, but they are not register in the cluster, cluster just says no instances available.
In ecs agent log on created instance i found logs flooded with one error:
Error registering: NoCredentialProviders: no valid providers in chain
The ec2 instances are created with a proper role ecs_role. This role has two policies, one of them is following, like docs required:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:CreateCluster",
"ecs:DeregisterContainerInstance",
"ecs:DiscoverPollEndpoint",
"ecs:Poll",
"ecs:RegisterContainerInstance",
"ecs:StartTelemetrySession",
"ecs:Submit*",
"ecs:StartTask"
],
"Resource": "*"
}
]
}
Im using ami ami-6ff4bd05. Latest terraform.
It was a problem with trust relationships in the role as the role should include ec2. Unfortunately the error message was not all that helpful.
Example of trust relationship:
{
"Version": "2008-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": ["ecs.amazonaws.com", "ec2.amazonaws.com"]
},
"Effect": "Allow"
}
]
}
Make sure you select the correct ECS role in the launch configuration.
You might want to add AmazonEC2RoleforSSM (or AmazonSSMFullAccess) to your EC2's role.
apparently this error message also occurs when an invalid aws-profile is passed.
I spent 2 days trying out everything without any luck. I have a standard setup i.e. ecs cluster instance in private subnet, ELB in public subnet, NAT and IGW properly set up in respective security groups, IAM role properly defined, standard config in NACL, etc. Despite everything the ec2 instances wouldnt register with the ecs cluster. Finally I figured out that my custom VPC's DHCP Options Set was configured for 'domain-name-servers: xx.xx.xx.xx, xx.xx.xx.xx' IP address of my org's internal DNS IPs...
The solution is to have following values for the DHCP Options Set:
Domain Name: us-west-2.compute.internal (assuming your vpc is in us-west-2),
Options: domain-name: us-west-2.compute.internal
domain-name-servers: AmazonProvidedDNS
I got this error today and figured out the problem: I missed setting the IAM role in launch template (it is under Advanced section). You need to set it to ecsInstanceRole (this is the default name AWS gives - so check if you have changed it and use accordingly).
I had switched from Launch Configuration to Launch Template, and while setting up the Launch Template, I missed adding the role!
if you use taskDefinition , check that you set execution & taskRole ARN's and set correct policies for that roles.
Related
I am learning AWS and I have the following task in an online training course:
Configure the MongoDB VM as highly privileged – configure an instance
profile to the VM and add the permission “ec2:*” as a custom policy.
I am trying to work out what that means. Is the task asking for a role that enables the VM instance to have full control over all EC2 resources?
If I understand it correctly, then I think the following policy would implement it.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:*"
],
"Effect": "Allow",
"Resource": "arn:aws:ec2:*:*:instance"
}
]
}
My understanding is that this policy is saying any EC2 instance can perform any EC2 action. Is that right?
I would say you are almost correct. Roles are attached to individual services which means your particular VM can perform any Ec2 action on this resource arn:aws:ec2:*:*:instance.
There is a difference in saying any ec2 can perform ec2 action instead that ec2 instance can perform any ec2 action to which this role is attached.
I created a EC2 role with the following permissions in cloudformation:
Policies:
- PolicyName: ec2-test-poilcy
PolicyDocument: |
{ "Version": "2012-10-17",
"Statement": [
{
"Action": [
"ssm:DescribeParameters"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Effect": "Allow",
"Action": [
"ssm:PutParameter",
"ssm:DeleteParameter",
"ssm:GetParameterHistory",
"ssm:GetParametersByPath",
"ssm:GetParameters",
"ssm:GetParameter",
"ssm:DeleteParameters"
],
"Resource": "arn:aws:ssm:REGION:ACCOUNT-ID:parameter/*"
}
]
}
And Attached the above role to my EC2 instance but, I was not able perform any SSM Parameter Store operations, I just get no response and after some couple of minutes, I get a timeout.
for example:
aws ssm get-parameter --name test --region us-east-2
Connect timeout on endpoint URL: "https://ssm.us-east-2.amazonaws.com/"
After trying various things, I attached the AdministratorAccess policy too, but it was still not responding to any ssm operations.
UPDATE
After spending hours, I realised that timeout error is only happening
for region: us-east-2
aws ssm get-parameter --name testing --region us-east-1
The above command ran successfully by using a different region .
Just to close my question as I found the problem, I am answering this myself, and hoping it helps someone else debugging.
I have used the below command for DNS info:
dig ssm.us-east-2.amazonaws.com
I got a response, In which I was able to observe that DNS is pointing to private IP:
;; ANSWER SECTION:
ssm.us-east-2.amazonaws.com. 60 IN A xx.xx.xx.xx7 (private ip)
But the correct DNS resolution would be something like this:
;; ANSWER SECTION:
ssm.us-east-2.amazonaws.com. 60 IN A 52.95.21.xxx
After spending some hours figuring out why, I found that it was due to the use of an unnecessary vpc endpoint for ssm service, So I removed it and now I am able to get a proper response for ssm operations for us-east-2 region.
Add HTTPS(443) Inbound Rule to security groups associated with VPC Endpoint allowing HTTPS(443) Inbound Rule for instance security group.
We can associate up to 5 Security Groups to VPC Endpoint.
If you're running your command from any EC2 instance, security group associated with the instance should be same as any security group associated with VPC endpoint, Or VPC endpoint associated security group should allow HTTPS(443) Inbound Rule to your instance security group.
I have an EKS cluster and EC2. I would like to create an instance profile and attach it to the EC2 - this profile should allow ONLY READ access to the EKS cluster.
Will the following policy be apt for this requirement?:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"eks:ListNodegroups",
"eks:DescribeFargateProfile",
"eks:ListTagsForResource",
"eks:ListAddons",
"eks:DescribeAddon",
"eks:ListFargateProfiles",
"eks:DescribeNodegroup",
"eks:ListUpdates",
"eks:DescribeUpdate",
"eks:AccessKubernetesApi",
"eks:DescribeCluster",
"eks:ListClusters",
"eks:DescribeAddonVersions"
],
"Resource": "*"
}
]
}
Depends on what you mean by "read" access.
If you mean "read" from AWS perspective as in being able to use the AWS CLI to tell you about EKS, yes that would be sufficient to get you started. This will not include any kubectl commands.
But if you mean read as in being able to execute kubectl commands against the cluster, then you will not be able to achieve that with this.
To implement read access to the cluster itself using kubectl commands, your EC2 instance will need a minimum of the following IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster",
"eks:ListClusters"
],
"Resource": "*"
}
]
}
With this, your EC2 will be able to execute eksctl utils write-kubeconfig --cluster=cluster-name to configure the kubeconfig. This also assumes you have the required components installed on your EC2 to run kubectl.
You also need to set up permissions within your cluster because the IAM permissions alone don't actually grant any visibility within the cluster itself.
The role you assign to your EC2 would need to be added to the aws-auth configmap in the kube-system namespace. See Managing users or IAM roles for your cluster from AWS docs for examples.
Unfortunately I don't believe there's a simple out-of-the-box RBAC role you can use that gives you read-only access to the entire cluster. Kubernetes provides four user-facing roles and of them, only the system:masters group has cluster-wide access.
Have a look at Kubernetes Using RBAC Authorization documentation - specifically on user-facing roles.
You will need to design a permission strategy to fit your needs, but you do have the default role view that you can start from. The default view user-facing role is tied to a ClusterRoleBinding and was designed / intended to be used in a namespace specific capacity.
Permissions and RBAC for Kubernetes is a very deep rabbit-hole.
I am following the aws documentation and created 3 security groups needed.
master
core
task
Now I am creating the cluster and it fails saying
Terminated with errorsAmazon EC2 permissions error when configuring required rules on securityGroup XXXXX.
What IAM permissions or other permissions I need to achieve this ?
Also how does a AWS service adds rules to the SG dynamically ? What permission does it need to achieve this ? Can it be more restrictive like the IAM roles/resources access?
Thanks,
If the service needs to modify security group rules, it must have the following permissions for the respective security groups:
"Effect": "Allow",
"Actions": [
"ec2:DescribeSecurityGroups",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress"
],
"Resources": [
"1st-security-group-arn",
"2nd-security-group-arn",
...
]
Note: Depending on what you're trying to do, it might also need "ec2:CreateSecurityGroup" and "ec2:DeleteSecurityGroup".
You can find the default permissions for the Amazon EMR Service Role in the documentation.
Context: I was going though Link to Amazon Web Services to create Swarms, in order to connect to my provider.
The role was created with success.
Then, while creating the policy, to associate to the role, a problem happened.
Problem:
An error occurred: Cannot exceed quota for PolicySize: 5120
As suggested by them, this is what I need to add in policy:
https://docs.docker.com/docker-for-aws/iam-permissions/
Did some research and people seem to like this solution:
https://github.com/docker/machine/issues/1655
How can I create the policy using the best method?
Noticing that the documentation in Docker is wrong - doesn't work in my case - what's the best method?
You are looking at the wrong instructions to connect docker-cloud to AWS follow these instructions: https://docs.docker.com/docker-cloud/infrastructure/link-aws/
It's the following 3 steps
Create AWS Policy for docker-cloud
Create a docker-cloud role and attache the policy from 1
Attach AWS role/account to docker-cloud
The policy in (1) above is pretty simple. It should be allowed to perform ec2 instances related actions (your screenshot of the policy looks like it doesn't provide ec2 permissions):
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:*",
"iam:ListInstanceProfiles"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
The role must have the permissions to implement the policy.
For a detailed post on the deployment via docker-cloud see: https://blog.geografia.com.au/how-we-are-using-docker-cloud-for-automated-testing-and-deployments-of-applications-bb87ec3173e7