Currently have configured AWS Fargate service with ApplicationLoadBalancedFargateService via AWS CDK(Python), would like to enable ExecuteCommand on the fargate containers to get access over them.
But currently unable to find a method to enable Exec on this fargate service.
Any help on this would be much appreciated.
In case anyone needs the command to enable shell access from the cli.
aws ecs update-service --cluster <cluster_name> --service <service_name> --task-definition <task_definition_name> --enable-execute-command --force-new-deployment
On a bit of a side note: You will also need to create a custom AWS Service: ecs-tasks role that includes Systems Manager permissions in addition to the AmazonECSTaskExecutionRolePolicy that comes in the default ecs-tasks role ecsTaskExecutionRole.
Here's an example policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:CreateControlChannel"
],
"Resource": "*"
}
]
}
One can now enable execute command via CDK:
declare const cluster: ecs.Cluster;
const loadBalancedFargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', {
cluster,
memoryLimitMiB: 1024,
desiredCount: 1,
cpu: 512,
taskImageOptions: {
image: ecs.ContainerImage.fromRegistry("amazon/amazon-ecs-sample"),
},
enableExecuteCommand: true
});
Source: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns-readme.html#ecs-exec
There seem to be an open issue/feature request in the aws-cdk repo that describes your issue.
The workaround is using Escape Hatches:
cfn_service = self.web_service.service.node.default_child
cfn_service.add_override("Properties.EnableExecuteCommand", True)
From my experience I learned that the ApplicationLoadBalancedFargateService is still limited. In some cases it makes sense to use the FargateService and create the loadbalancer, portmappings,... yourself.
Related
I have an EKS cluster and EC2. I would like to create an instance profile and attach it to the EC2 - this profile should allow ONLY READ access to the EKS cluster.
Will the following policy be apt for this requirement?:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"eks:ListNodegroups",
"eks:DescribeFargateProfile",
"eks:ListTagsForResource",
"eks:ListAddons",
"eks:DescribeAddon",
"eks:ListFargateProfiles",
"eks:DescribeNodegroup",
"eks:ListUpdates",
"eks:DescribeUpdate",
"eks:AccessKubernetesApi",
"eks:DescribeCluster",
"eks:ListClusters",
"eks:DescribeAddonVersions"
],
"Resource": "*"
}
]
}
Depends on what you mean by "read" access.
If you mean "read" from AWS perspective as in being able to use the AWS CLI to tell you about EKS, yes that would be sufficient to get you started. This will not include any kubectl commands.
But if you mean read as in being able to execute kubectl commands against the cluster, then you will not be able to achieve that with this.
To implement read access to the cluster itself using kubectl commands, your EC2 instance will need a minimum of the following IAM permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster",
"eks:ListClusters"
],
"Resource": "*"
}
]
}
With this, your EC2 will be able to execute eksctl utils write-kubeconfig --cluster=cluster-name to configure the kubeconfig. This also assumes you have the required components installed on your EC2 to run kubectl.
You also need to set up permissions within your cluster because the IAM permissions alone don't actually grant any visibility within the cluster itself.
The role you assign to your EC2 would need to be added to the aws-auth configmap in the kube-system namespace. See Managing users or IAM roles for your cluster from AWS docs for examples.
Unfortunately I don't believe there's a simple out-of-the-box RBAC role you can use that gives you read-only access to the entire cluster. Kubernetes provides four user-facing roles and of them, only the system:masters group has cluster-wide access.
Have a look at Kubernetes Using RBAC Authorization documentation - specifically on user-facing roles.
You will need to design a permission strategy to fit your needs, but you do have the default role view that you can start from. The default view user-facing role is tied to a ClusterRoleBinding and was designed / intended to be used in a namespace specific capacity.
Permissions and RBAC for Kubernetes is a very deep rabbit-hole.
I am setting up an AWS ECS Service using cloudformation and yaml syntax.
At some point, in the relevant documentation there is a property called Role whose definitions is the following:
Role
The name or ARN of an AWS Identity and Access Management (IAM) role that allows your Amazon ECS container agent to make calls to your
load balancer.
Note In some cases, you might need to add a dependency on the service role's policy. For more information, see IAM role policy in
DependsOn Attribute.
Required: No
Type: String
Update requires: Replacement
Since I intend to place the specific service behind an Application Load Balancer, is this property needed?
If so, do I need to create a new policy or are there any pre-defined policies that can serve this purpose?
Would it be enough if I just added the role/policy on the EC2 container instance level (e.g. append it in the relevant template that creates the ECS cluster offered by amazon?)
I would really appreciate any examples or use cases if any because the documentation is vague and incomplete on the topic.
You need the Role attribute if you want to use the Application Load balancer with your ECS service. As per the description the role allows your ECS service agent to connect to load balancer. If you are not using load balancer then, the field is optional.
Also setting the role on EC2 instance level is not needed. Since there are some policies which are related to ECS services, adding it at instance level role is not valid.
Please find below AWS managed polices to create the role:
{
"AttachedPolicies": [
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole",
"PolicyName": "AmazonEC2ContainerServiceAutoscaleRole"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
"PolicyName": "AmazonEC2ContainerServiceforEC2Role"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole",
"PolicyName": "AmazonEC2ContainerServiceRole"
}
]
}
Trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
To register your service with application load balancer you need the role for the service itself, but some permission needs for ECS container to register with cluster.
Amazon Elastic Container Service uses AWS Identity and Access
Management (IAM) service-linked roles. A service-linked role is a
unique type of IAM role that is linked directly to Amazon ECS.
Service-linked roles are predefined by Amazon ECS and include all the
permissions that the service requires to call other AWS services on
your behalf.
A service-linked role makes setting up Amazon ECS easier because you
don’t have to manually add the necessary permissions. Amazon ECS
defines the permissions of its service-linked roles, and unless
defined otherwise, only Amazon ECS can assume its roles. The defined
permissions include the trust policy and the permissions policy, and
that permissions policy cannot be attached to any other IAM entity.
So if you check this role this contain property
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
To Debug remove the role to service , it will not register.
If you create service using console by default it attach the role.
Amazon ECS needs permissions to register and deregister container
instances with your load balancer when tasks are created and stopped.
In most cases, the Amazon ECS service role is automatically created
for you in the Amazon ECS console first run experience. You can use
the following procedure to check and see if your account already has
an Amazon ECS service role.
This managed polici by AWS having ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Policy ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Description Default policy for Amazon ECS service role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
}
]
}
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/check-service-role.html
this is how its look like
If you do not assign a role to ECS container instance it will never show in you ECS cluster and it will throw an error
2018-09-06T15:26:22Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
under /var/logs/ecs
tail -f ecs-agent.log.2018-09-06
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html
If I am running container in AWS ECS using EC2, then I can access running container and execute any command.
ie.
docker exec -it <containerid> <command>
How can I run commands in the running container or access container in AWS ECS using Fargate?
Update(16 March, 2021):
AWS announced a new feature called ECS Exec which provides the ability to exec into a running container on Fargate or even those running on EC2. This feature makes use of AWS Systems Manager(SSM) to establish a secure channel between the client and the target container. This detailed blog post from Amazon describes how to use this feature along with all the prerequisites and the configuration steps.
Original Answer:
With Fargate you don't get access to the underlying infrastructure so docker exec doesn't seem possible. The documentation doesn't mention this explicitly but it's mentioned in this Deep Dive into AWS Fargate presentation by Amazon where this is mentioned on slide 19:
Some caveats: can’t exec into the container, or access the underlying
host (this is also a good thing)
There's also some discussion about it on this open issue in ECS CLI github project.
You could try to run an SSH server inside a container to get access but I haven't tried it or come across anyone doing this. It also doesn't seem like a good approach so you are limited there.
AWS Fargate is a managed service and it makes sense not to allow access into containers.
If you need to troubleshoot the container you can always increase the log level of your app running in containers. Best practices on working with containers says
"Docker containers are in fact immutable. This means that a running
container never changes because in case you need to update it, the
best practice is to create a new container with the updated version of
your application and delete the old one."
Hope it helps.
You need to provide a "Task role" for a Task Definition (this is different than the "Task execution role"). This can be done by first going to IAM
IAM role creation
IAM > roles > create role
custom trust policy > copy + paste
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Add permission > Create Policy
JSON > replace YOUR_REGION_HERE & YOUR_ACCOUNT_ID_HERE & CLUSTER_NAME > copy + paste
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:YOUR_REGION_HERE:YOUR_ACCOUNT_ID_HERE:log-group:/aws/ecs/CLUSTER_NAME:*"
}
]
}
Give it a name
go back to Add permissions > search by name > check > Next
Give a role name > create role
ECS new task
go back to ECS > go to task definition and create a new revision
select your new role for "Task role" (different than "Task execution role") > update Task definition
go to your service > update > ensure revision is set to latest > finish update of the service
current task and it should auto provision your new task with its new role.
try again
Commands I used to exec in
enables execute command
aws ecs update-service --cluster CLUSTER_NAME --service SERVICE_NAME --region REGION --enable-execute-command --force-new-deployment
adds ARN to environment for easier cli. Does assume only 1 task running for the service, otherwise just manually go to ECS and grab arn and set them for your cli
TASK_ARN=$(aws ecs list-tasks --cluster CLUSTER_NAME --service SERVICE_NAME --region REGION --output text --query 'taskArns[0]')
see the task,
aws ecs describe-tasks --cluster CLUSTER_NAME --region REGION --tasks $TASK_ARN
exec in
aws ecs execute-command --region REGION --cluster CLUSTER_NAME --task $TASK_ARN --container CONTAINER --command "sh" --interactive
As on 16 March 2021, AWS has introduced ECS Exec which can be used to run command on container running in either EC2 or Fargate.
URL will be available at
https://aws.amazon.com/about-aws/whats-new/2021/03/amazon-ecs-now-allows-you-to-execute-commands-in-a-container-running-on-amazon-ec2-or-aws-fargate/
Context: I was going though Link to Amazon Web Services to create Swarms, in order to connect to my provider.
The role was created with success.
Then, while creating the policy, to associate to the role, a problem happened.
Problem:
An error occurred: Cannot exceed quota for PolicySize: 5120
As suggested by them, this is what I need to add in policy:
https://docs.docker.com/docker-for-aws/iam-permissions/
Did some research and people seem to like this solution:
https://github.com/docker/machine/issues/1655
How can I create the policy using the best method?
Noticing that the documentation in Docker is wrong - doesn't work in my case - what's the best method?
You are looking at the wrong instructions to connect docker-cloud to AWS follow these instructions: https://docs.docker.com/docker-cloud/infrastructure/link-aws/
It's the following 3 steps
Create AWS Policy for docker-cloud
Create a docker-cloud role and attache the policy from 1
Attach AWS role/account to docker-cloud
The policy in (1) above is pretty simple. It should be allowed to perform ec2 instances related actions (your screenshot of the policy looks like it doesn't provide ec2 permissions):
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:*",
"iam:ListInstanceProfiles"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
The role must have the permissions to implement the policy.
For a detailed post on the deployment via docker-cloud see: https://blog.geografia.com.au/how-we-are-using-docker-cloud-for-automated-testing-and-deployments-of-applications-bb87ec3173e7
I have files stored on S3 and wrote .ebextensions config to automatically copy the them to new instances. I'm receiving this error in the Elastic Beanstalk console:
[Instance: INSTANCEID Module: AWSEBAutoScalingGroup ConfigSet: null] Command failed on instance. Return code: 1 Output: [CMD-AppDeploy/AppDeployStage0/EbExtensionPreBuild] command failed with error code 1: Error occurred during build: Failed to retrieve https://s3-us-west-1.amazonaws.com/MyBucket/MyFolder/_MyFile.txt: HTTP Error 403 : AccessDenied
My .ebextension config file has this section:
files:
"/target/file/path" :
mode: "000777"
owner: ec2-user
group: ec2-user
source: https://s3-us-west-1.amazonaws.com/_MyBucket_/_MyFolder_/_MyFile.txt
In attempting to make this file copying work, I've also relaxed permissions by giving the elastic beanstalk IAM role the standard read only access policy to all of S3. It's policy is this:
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
Yet the prebuild copying step still fails. Did I give the source url in the correct format? Is there another security entity/policy involved? Help please :)
The documentation is very sketchy on the subject (probably an ideal candidate for StackExchange Docs!).
To do this correctly with .ebextensions, you need to allow the Beanstalk instance IAMs user in the bucket policy, setup an AWS::CloudFormation::Authentication: auth config and attach config to remote sources. This is kind of a hybrid of all the other answers, but all failed in one way or another for me.
Assuming your IAM instance role is aws-elasticbeanstalk-ec2-role:
Set your AWS bucket to allow the Beanstalk IAM User. Edit "bucket policy":
{
"Version": "2012-10-17",
"Id": "BeanstalkS3Copy",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "<beanstalk_iam_role_arn>"
},
"Action": [
"s3:ListBucketVersions",
"s3:ListBucket",
"s3:GetObjectVersion",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<bucket_name>",
"arn:aws:s3:::<bucket_name>/*"
]
}
]
}
where:
beanstalk_iam_role_arn = the fully qualified instance IAMs role. See "IAM role" associated with a running instance if available or see environment configuration. Example: arn:aws:iam::12345689:role/aws-elasticbeanstalk-ec2-role
bucket_name = your bucket name
In your .ebextension/myconfig.config, add an S3 authentication block that uses your IAMs instance user:
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Auth:
type: "s3"
buckets: ["bucket_name"]
roleName:
"Fn::GetOptionSetting":
Namespace: "aws:asg:launchconfiguration"
OptionName: "IamInstanceProfile"
DefaultValue: "aws-elasticbeanstalk-ec2-role"
Set bucket_name appropriately
Define a remote file and attach the S3 Authentication block:
"/etc/myfile.txt" :
mode: "000400"
owner: root
group: root
authentication: "S3Auth" # Matches to auth block above.
source: https://s3-eu-west-1.amazonaws.com/mybucket/myfile.txt
Set your source URL appropriately
Similar to chaseadamsio's answer, you can configure the role given to the EC2 instance with a policy to access S3 resources, then use the pre-installed AWS CLI utilities to move files around.
The way I approached this is to create a role dedicated to the given EB application, then attach a policy similar to:
"Statement": [
{
"Sid": "<sid>",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<your_bucket_path>/*"
]
}
]
This gives your instance access, then to get the files, add a 'commands' block to your config such as:
commands:
01-get-file:
command: aws s3 cp s3://<your_bucket_path>/your-file.txt /home/ec2-user
02-execute-actions:
[unpack, run scripts, etc..]
Obviously you can use other AWS CLI utlities as needed. I found this solved a lot of problems I was having with S3 access and makes deployment a lot easier.
I found a solution to overcome this error. It turns out adding a Resources section to the .ebextensions config file makes it work. The entire file becomes:
files:
"/target/file/path" :
mode: "000777"
owner: ec2-user
group: ec2-user
source: https://s3-us-west-1.amazonaws.com/_MyBucket_/_MyFolder_/_MyFile.txt
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
type: S3
roleName: aws-elasticbeanstalk-ec2-role
buckets: _MyBucket
At this point, I don't know enough to grok why it has to be this way. Hopefully it can help someone who's lost move forward and eventually gain a better understanding. I based my answer on this link https://forums.aws.amazon.com/message.jspa?messageID=541634
An alternative to setting the .ebextensions config would be to set a policy on the aws-elasticbeanstalk-ec2-role within the IAM Manager (or create a new role specifically for your elastic beanstalk environments to sandbox your autoscaled ec2 instances.
To do so, go to the IAM manager within the web console, and click on "Roles" on the left side. You should see your instance name in the list of roles, clicking on that will take you to the administration page for that particular role. Attach a new role policy to the role under "Permissions" with a policy document matching what you want your ec2 to have permissions to do (in this case, you'd give it a policy to access an s3 bucket called _MyBucket and you should no longer need the Resources section in your .ebextensions config.
If you have your IAM role for the machine configured to get access to the file you can do the following in .ebextensions
commands:
01a_copy_file:
command: aws s3 cp s3://bucket/path/file /destination/