I have an ECS service which is of Launch Type EC2 owned by an AWS account A. Our IT team has created an FSx storage owned by an AWS Account B:
When I try to launch tasks I get this not authorized error in the Stopped reason section of the task:
Fsx describing filesystem(s) from the service for [fs-0fd8b05f434cf0e72]:
FileSystemNotFound: File system 'fs-0fd8b05f434cf0e72' does not exist.
I have attached those 2 policies to the EC2 (container host) instance:
AmazonFSxReadOnlyAccess (AWS Managed)
fsx_mount (Customer Managed)
fsx_mount:
{
"Statement": [
{
"Action": [
"secretsmanager:GetSecretValue"
],
"Effect": "Allow",
"Resource": "arn:aws:secretsmanager:us-west-2:111111111111:secret:dev/rushmore/ad-account-NKOkyh"
},
{
"Action": [
"fsx:*",
"ds:DescribeDirectories"
],
"Effect": "Allow",
"Resource": "arn:aws:fsx:eu-west-1:222222222222:file-system/fs-0fd8b05f434cf0e72"
}
],
"Version": "2012-10-17"
}
Note that the account id of 222222222222 represents AWS Account B.
Terraform aws_ecs_task_definition:
resource "aws_ecs_task_definition" "participants_task" {
volume {
name = "FSxStorage"
fsx_windows_file_server_volume_configuration {
file_system_id = "fs-0fd8b05f434cf0e72"
root_directory = "\\data"
authorization_config {
credentials_parameter = aws_secretsmanager_secret_version.fsx_account_secret.arn
domain = var.domain
}
}
}
...
}
I am not sure why ECS cannot "see" the FSx file system. Surely it must be because it is in another AWS account but I don't know what changes are required in order to fix this.
From AWS documentation:
You can access your FSx for Windows File Server file system from
compute instances in a different VPC, AWS account, or AWS Region from
that associated with your file system. To do so, you can use VPC
peering or transit gateways. When you use a VPC peering connection or
transit gateway to connect VPCs, compute instances that are in one VPC
can access Amazon FSx file systems in another VPC. This access is
possible even if the VPCs belong to different accounts, and even if
the VPCs reside in different AWS Regions.
The short version of the above text is that your ECS service and Amazon FSx Windows File server either need to be in the same VPC or need to be in VPCs which are connected to each other (via VPC peering or Transit Gateway).
Related
Is there a way where to attach an AWS IAM profile to an Azure VM.
I'm trying to develop a common infrastructure for Azure and AWS and i want to use resources which are in AWS from an Azure VM.
I know this can do this by exporting AWS creds to Azure VM but is there a way where I can attach an already existing AWS IAM profile to the Azure VM (if not directly may be through an interface or a service?) and access the resources (which how is I'm doing from an ec2 instance currently) ?
Sadly you can't do this. IAM instance profiles are only valid and usable from ec2 instances. You can't use instance profiles from outside of aws.
As you mentioned, you have to explicitly provide aws credentials to your azure vm. For example by creating .aws/ folder with aws profile.
You should be able to achieve what you are looking for by using the same IAM role for both the EC2 instance profile and a managed identity assigned to your Azure VM.
From my limited understanding of AWS, the instance profile identifies your EC2 instance, so you cant use it directly.
To use the same role for both EC2 and Azure VM, here is how I would try this out:
First, familiarize yourself with how you can assign an Azure managed identity a role in AWS. I wrote this blog post recently to show how Managed identities can be granted access to AWS resources: https://blog.identitydigest.com/azuread-access-aws/
Now rather than create a new role as mentioned in the blog, you can reuse the role in your EC2 instance, by adding a trust relationship for the managed identity to AssumeRoleWithWebIdentity.
So the trust relationship for your existing role used in your EC2 profile will look something as follows: (please note, I have not tried multiple statements in a role but expect this to work based on the AWS documentation)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<your account>:oidc-provider/sts.windows.net/<your azure ad tenant>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"sts.windows.net/<your tenant>:aud": "app audience or managed identity client_id",
"sts.windows.net/<your tenant>:sub": "in case you want to also include sub"
}
}
}
]
}
Now if you assign the managed identity to the VM, it should be able to access the same resources as your EC2 instance.
I am running a K8S cluster in AWS with EKS and external-DNS with an ingress-Nginx-ingress-controller. Application is working fine and accessible on load-balancer. Now I am trying to point my DNS to this cluster i.e. e1.exapmle.com and for this, I am following below documentation.
https://github.com/kubernetes-sigs/external-dns
The issue I am facing is that I have two AWS accounts (QA, Staging) and hosted zones are created in staging account and the cluster is running on QA account. I have tried cross-account IAM roles for this to work but I can only see hosted zones via the web console, cli or external-DNS pods are not able to able to communicate with Route53.
This is what I have done so far:
Create an IAM policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets"
],
"Resource": [
"arn:aws:route53:::hostedzone/*"
]
},
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones",
"route53:ListResourceRecordSets"
],
"Resource": [
"*"
]
}
]
}
Create a cross-account IAM role trusting account staging and trusted account QA.
Assum policy in the trusted account by specifying IAM role ARN for the above policy.
By doing above I only managed to see hosted zones via web console when I switch roles.
I assume you provided a role arn of a role in the EKS account to the extern-dns pods which is allowed to assume to role inside you HostedZone account!?
This will not work because then (as you do inside the console) you need to explicitly assume the role with existing permissions.
You need to pass the Cross-Account role with your attached route53 policy directly to the external-dns pods.
Pod -> assumes role in different account -> receives temp. credentials
pod -> assumed role in eks account (-> allowd to assume cross-accout role)
last step won’t happen i guess
I have two VPC in my account. One for Test and other for Prod environment.
I am trying to setup IAM user accounts for developers, with permission boundaries, so that developers only has access to create/modify resources in Test VPC.
How do I do that? Can you share a sample policy JSON?
An Amazon VPC is a virtual network.
It is not possible to control access to a network based on "users" because the network has no knowledge of users. It can only control traffic by IP address and protocol.
If you want developers to be able to login to instances in Test environment, but not a Prod environment, you would either need to control access on the instances themselves (eg when they login to an EC2 instance), or control access to the network (eg by controlling access to a VPN connection or having developers access resources on a network with a known IP address range).
This is exactly the same as controlling access on a corporate network — developers could be placed on a network that has access to Test resources, while Sys Admins could be placed on a network that has access to Prod resources. This has to do with how their computer is connected to the network, rather than "who" they are.
If, instead, your goal is to restrict the Dev's ability to create/change resources in a VPC, then this can be done by adding conditions a IAM policies. For example, granting them the ability to launch an EC2 instance but only in the Test VPC.
See: How to Help Lock Down a User’s Amazon EC2 Capabilities to a Single VPC | AWS Security Blog
What I get is you are trying to restrict users to the services which are under a particular VPC. I did the same thing for allowing users to update Lambda functions which are inside a particular VPC only. This can be done like below:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAllResources",
"Effect": "Allow",
"Action": "*",
"Resource": "*"
},
{
"Sid": "DenyLambdaUpdatIfNotInsideVPC",
"Effect": "Deny",
"Action": [
"lambda:CreateFunction",
"lambda:UpdateFunctionConfiguration"
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"lambda:VpcIds": "your vpc id"
}
}
}
]
}
In this way you can restrict users from accessing the resources which are outside your VPC by writing services and their specific actions in the deny statement.
I am setting up an AWS ECS Service using cloudformation and yaml syntax.
At some point, in the relevant documentation there is a property called Role whose definitions is the following:
Role
The name or ARN of an AWS Identity and Access Management (IAM) role that allows your Amazon ECS container agent to make calls to your
load balancer.
Note In some cases, you might need to add a dependency on the service role's policy. For more information, see IAM role policy in
DependsOn Attribute.
Required: No
Type: String
Update requires: Replacement
Since I intend to place the specific service behind an Application Load Balancer, is this property needed?
If so, do I need to create a new policy or are there any pre-defined policies that can serve this purpose?
Would it be enough if I just added the role/policy on the EC2 container instance level (e.g. append it in the relevant template that creates the ECS cluster offered by amazon?)
I would really appreciate any examples or use cases if any because the documentation is vague and incomplete on the topic.
You need the Role attribute if you want to use the Application Load balancer with your ECS service. As per the description the role allows your ECS service agent to connect to load balancer. If you are not using load balancer then, the field is optional.
Also setting the role on EC2 instance level is not needed. Since there are some policies which are related to ECS services, adding it at instance level role is not valid.
Please find below AWS managed polices to create the role:
{
"AttachedPolicies": [
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceAutoscaleRole",
"PolicyName": "AmazonEC2ContainerServiceAutoscaleRole"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
"PolicyName": "AmazonEC2ContainerServiceforEC2Role"
},
{
"PolicyArn": "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole",
"PolicyName": "AmazonEC2ContainerServiceRole"
}
]
}
Trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
To register your service with application load balancer you need the role for the service itself, but some permission needs for ECS container to register with cluster.
Amazon Elastic Container Service uses AWS Identity and Access
Management (IAM) service-linked roles. A service-linked role is a
unique type of IAM role that is linked directly to Amazon ECS.
Service-linked roles are predefined by Amazon ECS and include all the
permissions that the service requires to call other AWS services on
your behalf.
A service-linked role makes setting up Amazon ECS easier because you
don’t have to manually add the necessary permissions. Amazon ECS
defines the permissions of its service-linked roles, and unless
defined otherwise, only Amazon ECS can assume its roles. The defined
permissions include the trust policy and the permissions policy, and
that permissions policy cannot be attached to any other IAM entity.
So if you check this role this contain property
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
To Debug remove the role to service , it will not register.
If you create service using console by default it attach the role.
Amazon ECS needs permissions to register and deregister container
instances with your load balancer when tasks are created and stopped.
In most cases, the Amazon ECS service role is automatically created
for you in the Amazon ECS console first run experience. You can use
the following procedure to check and see if your account already has
an Amazon ECS service role.
This managed polici by AWS having ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Policy ARN
arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole
Description Default policy for Amazon ECS service role.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:Describe*",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:Describe*",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"elasticloadbalancing:RegisterTargets"
],
"Resource": "*"
}
]
}
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/check-service-role.html
this is how its look like
If you do not assign a role to ECS container instance it will never show in you ECS cluster and it will throw an error
2018-09-06T15:26:22Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
under /var/logs/ecs
tail -f ecs-agent.log.2018-09-06
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html
I have followed the documentation for spinning up an EKS cluster that says to make a service role with certain policies.
https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf
To create your Amazon EKS service role
1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. Choose Roles, then Create role.
3. Choose EKS from the list of services, then Allows Amazon EKS to manage your clusters on your behalf for your use case, then Next: Permissions.
4. Choose Next: Review.
5. For Role name, enter a unique name for your role, such as eksServiceRole, then choose Create role.
When I create a basic hello world app, it throws an AccessDenied error.
Error creating load balancer (will retry): failed to ensure load balancer for service default/nginx:
AccessDenied: User: arn:aws:sts::*************:assumed-role/eks-service-role/************* is not authorized to perform: iam:CreateServiceLinkedRole on resource: arn:aws:iam::*************:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
The two Policies that were added (AmazonEKSClusterPolicy, AmazonEKSServicePolicy) do not have the iam:CreateServiceLinkedRole action allowed. Are we supposed to add this outside of the policies defined in the guide? Or is this something that should be included in the EKS policies?
It seems that the EKS userguide assumes you have created load balancers in your AWS account prior to creating the EKS cluster, and thus have an existing AWSServiceRoleForElasticLoadBalancing service role in AWS IAM.
As described in https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/elb-service-linked-roles.html#create-service-linked-role
You don't need to manually create the AWSServiceRoleForElasticLoadBalancing role. Elastic Load Balancing creates this role for you when you create a load balancer.
EKS is attempting to do this for you, resulting in the access denied exception using the default policies.
Other options to explicitly create the service-linked role prior to EKS cluster creation include:
AWS CLI
aws iam create-service-linked-role --aws-service-name "elasticloadbalancing.amazonaws.com"
Terraform
resource "aws_iam_service_linked_role" "elasticloadbalancing" {
aws_service_name = "elasticloadbalancing.amazonaws.com"
}
Or, manually create a load balancer from the UI Console.
Regardless of provisioning options, you should know things will work when you see the following role in AWS IAM
arn:aws:iam::<ACCOUNT_ID>:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing
I got it worked by adding this policy to the EKS Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAccountAttributes"
],
"Resource": "*"
}
]
}