IAM Error while using ecs-cli - amazon-web-services

I'm trying to create a new Task for ECS using a compose file, but i'm getting an AccessDeniedException even when my user has the required permissions.
$ ecs-cli compose --project-name test create
WARN[0000] Skipping unsupported YAML option for service... option name=build service name=builder
WARN[0000] Skipping unsupported YAML option for service... option name=restart service name=db
WARN[0000] Skipping unsupported YAML option for service... option name=restart service name=dbadmin
WARN[0000] Skipping unsupported YAML option for service... option name=restart service name=app
ERRO[0001] Error registering task definition error=AccessDeniedException: User: arn:aws:iam::XXXXXXX:user/foo is not authorized to perform: ecs:RegisterTaskDefinition on resource: *
status code: 400, request id: 41e6b69a-a839-11e6-84b0-e9bc2ec3f81b family=ecscompose-test
ERRO[0001] Create task definition failed error=AccessDeniedException: User: arn:aws:iam::XXXXXXX:user/foo is not authorized to perform: ecs:RegisterTaskDefinition on resource: *
status code: 400, request id: 41e6b69a-a839-11e6-84b0-e9bc2ec3f81b
FATA[0001] AccessDeniedException: User: arn:aws:iam::XXXXXXX:user/foo is not authorized to perform: ecs:RegisterTaskDefinition on resource: *
status code: 400, request id: 41e6b69a-a839-11e6-84b0-e9bc2ec3f81b
The user have this policy attached:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecs:RegisterTaskDefinition",
"ecs:ListTaskDefinitions",
"ecs:DescribeTaskDefinition"
],
"Resource": [
"*"
]
}
]
}
I also tried attaching the AmazonEC2ContainerServiceFullAccess (that have ecs:*), but didn't work.

Found the problem, the user i was using had a policy to use MFA (MultiFactor Auth), that is not supported by the ecs-cli.

I believe this posting has some answers as to why the above error is happening, thought not a fix.
Trouble deploying docker on AWS with ecs-cli
"From what I understand, ecs-cli has a very limited support of the complete Docker Compose file syntax"
per user Dolan Antenucci
Note the warnings
"WARN[0000] Skipping unsupported YAML option for service..."

ECS does not support a big chunk of the compose settings. However, it should just print warnings and ignore them, which will produce unintended results, but should not be throwing permission issues.
When you see 400 AccessDeniedExceptions that are in the form of "user_arn not authorized to perform service:action on service_resource" it is definitely an IAM issue. However, the IAM policy you listed looks correct. My thinking is that you are somehow not using the correct user credentials, or that the IAM policy is not applied correctly to the user.

Related

Deploying AWS Lambda "Hello World" using sam init, sam build sam deploy defaults fails with Failed to create changeset for sam-app

I have followed the simplest route to create a basic AWS lambda function on my PC - but when I try to deploy it it fails with an obscure message.
Error: Failed to create changeset for the stack: sam-app, An error occurred (InternalFailure) when calling the CreateChangeSet operation (reached max retries: 4): Unknown
I believe I have provided all the required permissions :
AmazonS3FullAccess
AWSCodeDeployFullAccess
AWSLambdaRole
AWSCloudFormationFullAccess
AWSLambda_FullAccess
and additionally :
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"iam:DetachRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:CreateRole"
],
"Resource": "*"
}
]
}
My terminal session commands & responses are :
c:\My Projects\Nuzum-AWS\sam-app>sam deploy --guided --template-file .aws-sam\build\template.yaml
Configuring SAM deploy
Looking for config file [samconfig.toml] : Not found
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]:
AWS Region [eu-west-1]:
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]:
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]:
#Preserves the state of previously provisioned resources when an operation fails
Disable rollback [y/N]:
HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
Save arguments to configuration file [Y/n]:
SAM configuration file [samconfig.toml]:
SAM configuration environment [default]:
Looking for resources needed for deployment:
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-dee0ker00dts
A different default S3 bucket can be set in samconfig.toml
Saved arguments to config file
Running 'sam deploy' for future deployments will use the parameters saved above.
The above parameters can be changed by modifying samconfig.toml
Learn more about samconfig.toml syntax at
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Uploading to sam-app/7b33e513850748d9482c8e504aacefb2 533 / 533 (100.00%)
Deploying with following values
===============================
Stack name : sam-app
Region : eu-west-1
Confirm changeset : False
Disable rollback : False
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-dee0ker00dts
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Signing Profiles : {}
Initiating deployment
Uploading to sam-app/4e272b888be3f30dedd7ac0cf77b1925.template 1361 / 1361 (100.00%)
Error: Failed to create changeset for the stack: sam-app, An error occurred (InternalFailure) when calling the CreateChangeSet operation (reached max retries: 4): Unknown
You gave confirm change set value as false. you have to give that as true.
Can you try giving these values while deploying?
Confirm changes before deploy [y/N]:y.
Allow SAM CLI IAM role creation [Y/n]:y
Disable rollback [y/N]:n
Save arguments to configuration file [Y/n]:y

Not authorized to perform: arsenal:RegisterOnPremisesAgent - no identity-based policy allows the arsenal:RegisterOnPremisesAgent

After installing the AWS Application Discovery Agent on a Ubuntu Server the agent daemon logs are failing with the following message:
2022-08-20 03:23:54 info [0xf71a4d88] Attempting registration for on premises agent with agentId: and agentVersion: 2.0.2042.0
2022-08-20 03:23:54 info [0xf71a4d88] Arsenal endpoint is "https://arsenal-discovery.us-east-1.amazonaws.com".
2022-08-20 03:23:55 info [0xf71a4d88] Registration failed.; User: arn:aws:iam::000000000000:user/Migrator is not authorized to perform: arsenal:RegisterOnPremisesAgent on resource: * because no identity-based policy allows the arsenal:RegisterOnPremisesAgent action( AccessDeniedException = 15 ) at ArsenalController.cpp; UnpackRegisterResult; line 321
2022-08-20 03:23:55 info [0xf71a4d88] Failure communicating with Arsenal.
2022-08-20 03:23:55 info [0xf71a4d88] Caught an exception of type 13CAWSException; what= Registration failed.; User: arn:aws:iam::000000000000:user/Migrator is not authorized to perform: arsenal:RegisterOnPremisesAgent on resource: * because no identity-based policy allows the arsenal:RegisterOnPremisesAgent action( AccessDeniedException = 15 ) at ArsenalController.cpp; UnpackRegisterResult; line 321; caught at ArsenalController.cpp; CallArsenal; line 885
Such logs are kept at /var/log/aws/discovery/ within the Ubuntu Server in which the agent is installed. I'll provide details and steps on how I'm getting into this error.
Following AWS Application Discovery documentation, I've attached these managed policies to my migration role:
AWSApplicationDiscoveryServiceFullAccess
AWSApplicationDiscoveryAgentAccess
Here are the commands I'm using:
aws iam create-user --user-name 'Migrator'
aws iam create-role --role-name 'MigrationRole' --assume-role-policy-document 'file://trust-policy.json'
aws iam attach-role-policy --policy-arn 'arn:aws:iam::aws:policy/AWSApplicationDiscoveryServiceFullAccess' --role-name 'MigrationRole'
aws iam attach-role-policy --policy-arn 'arn:aws:iam::aws:policy/AWSApplicationDiscoveryAgentAccess' --role-name 'MigrationRole'
The trust relationship document trust-policy.json is this one:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::000000000000:user/Migrator"
},
"Action": "sts:AssumeRole"
}
]
}
The documentation is clear in which the managed policy AWSApplicationDiscoveryAgentAccess adds permissions to Arsenal agent registration:
This policy also grants the user access to Arsenal. Arsenal is an agent service that is managed and hosted by AWS. Arsenal forwards data to Application Discovery Service in the cloud.
Indeed, we can confirm looking in the policy associations attached to my role that permission arsenal:RegisterOnPremisesAgent is allowed:
I'm installing the agent using these commands, also referenced from the documentation:
sudo apt update
sudo apt upgrade -y
curl -o ./aws-discovery-agent.tar.gz https://s3-us-west-2.amazonaws.com/aws-discovery-agent.us-west-2/linux/latest/aws-discovery-agent.tar.gz
tar -xzf aws-discovery-agent.tar.gz
sudo bash install -r us-east-1 -k ABCDEFABCDEFABCDEFAB -s ABCDEFABCDEFABCDEFABCDEFABCDEFABCDEFABCD
For sanity check I've added AdministratorAccess to the role but the problem persists.
Ok just figured this one out.
I was doing an incorrect attachment to the role which was not delegating the permissions to the user identity.
I switched to a group attachment and the agent started to work immediately:
aws iam create-group --group-name MigrationGroup
aws iam add-user-to-group --user-name Migrator --group-name MigrationGroup
aws iam attach-group-policy --policy-arn 'arn:aws:iam::aws:policy/AWSApplicationDiscoveryServiceFullAccess' --group-name 'MigrationGroup'
aws iam attach-group-policy --policy-arn 'arn:aws:iam::aws:policy/AWSApplicationDiscoveryAgentAccess' --group-name 'MigrationGroup'
And the permissions are now visible in the user registry:

AWS X-RAY [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated

I'm using X-Ray Daemon with Fargate. Unfortunately in the logs of the daemon I see errors like:
[Error] Sending segment batch failed with: NoCredentialProviders: no
valid providers in chain. Deprecated. For verbose messaging see
aws.Config.CredentialsChainVerboseErrors Warn] Delaying sending of
additional batches by 0 seconds
AWS X-Ray works with IAM: permission: AWSXrayFullAccess
I already checked:
https://github.com/aws-samples/aws-xray-fargate
https://docs.amazonaws.cn/en_us/xray/latest/devguide/xray-guide.pdf
{
"name" : "xray-daemon",
"image" : "amazon/aws-xray-daemon",
"user" : "1337",
"essential" : true,
"cpu" : "32",
"memoryReservation" : "256",
"portMappings" : [
{
"containerPort" : 2000,
"protocol" : "udp"
}
]
}
],
"requiresCompatibilities" : [ "FARGATE" ],
"taskRoleArn" : "arn:aws:iam::123456789012:role/ecsTaskRole",
"executionRoleArn" : "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
"networkMode" : "awsvpc"
}
You should check you role permissions role, I used the same for execution & taskRoleArn. my role contained these policies
arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
arn:aws:iam::aws:policy/AWSXRayFullAccess
and also check trust relationships for role. my used:
ecs-tasks.amazonaws.com
ecs.amazonaws.com
c2.amazonaws.com
I got this error when my version of the x-ray deamon was old.
Follow this as guidance to update your x-ray deamon if it is running on ECS:
https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html
XRay daemon is missing credentials needed to authenticate with XRay server. You can pass credentials to XRay daemon in several ways, but the most common are:
through environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
through ~/.aws/credentials file (see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
I'm guessing option 1 will be easier with fargate.
Of course you need to generate the key in IAM, see: https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html
The user you'll create the access key for will need to have permissions to use XRay (AWSXrayFullAccess should do).

AWS Glue Job getting Access Denied when writing to S3

I have a Glue ETL job, created by CloudFormation. This job extracts data from RDS Aurora and write to S3.
When I run this job, I get the error below.
The job has an IAM service role.
This service role allows
Glue and RDS service,
assume arn:aws:iam::aws:policy/AmazonS3FullAccess and arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole, and
has full range of rds:* , kms:* , and s3:* actions allow to the corresponding RDS, KMS, and S3 resources.
I have the same error whether the S3 bucket is encrypted with either AES256 or aws:kms.
I get the same error whether the job has a Security Configuration or not.
I have a job doing the exactly same thing that I created manually and can run successfully without a Security Configuration.
What am I missing? Here's the full error log
"/mnt/yarn/usercache/root/appcache/application_1...5_0002/container_15...45_0002_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o145.pyWriteDynamicFrame.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 2.0 failed 4 times, most recent failure: Lost task 3.3 in stage 2.0 (TID 30, ip-10-....us-west-2.compute.internal, executor 1): com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: F...49), S3 Extended Request ID: eo...wXZw=
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588
Unfortunately the error doesn't tell us much except that it's failing during the write of your DynamicFrame.
There is only a handful of possible reasons for the 403, you can check if you have met them all:
Bucket Policy rules on the destination bucket.
The IAM Role needs permissions (although you mention having S3*)
If this is cross-account, then there is more to check with regards things like to allow-policies on the bucket and user. (In general a Trust for the Canonical Account ID is simplest)
I don't know how complicated your policy documents might be for the Role and Bucket, but remember that an explicit Deny statement takes precedence over an allow.
If the issue is KMS related, I would check to ensure your Subnet you select for the Glue Connection has a route to reach the KMS endpoints (You can add an Endpoint for KMS in VPC)
Make sure issue is not with the Temporary Directory that is also configured for your job or perhaps write-operations that are not your final.
Check that your account is the "object owner" of the location you are writing to (normally an issue when read/writing data between accounts)
If none of the above works, you can shed some more light with regards to your setup. Perhaps the code for write-operation.
In addition to Lydon's answer, error 403 is also received if your Data Source location is the same as the Data Target; defined when creating a Job in Glue. Change either of these if they are identical and the issue will be resolved.
You should add a Security configurations(mentioned under Secuity tab on Glue Console). providing S3 Encryption mode either SSE-KMS or SSE-S3.
Security Configuration
Now select the above security configuration while creating your job under Advance Properties.
Duly verify you IAM role & S3 bucket policy.
It will work
How are you providing permission for PassRole to glue role?
{
"Sid": "AllowAccessToRoleOnly",
"Effect": "Allow",
"Action": [
"iam:PassRole",
"iam:GetRole",
"iam:GetRolePolicy",
"iam:ListRolePolicies",
"iam:ListAttachedRolePolicies"
],
"Resource": "arn:aws:iam::*:role/<role>"
}
Usually we create roles using <project>-<role>-<env> e.g. xyz-glue-dev where project name is xyz and env is dev. In that case we use "Resource": "arn:aws:iam::*:role/xyz-*-dev"
For me it was two things.
Access policy for a bucket should be given correctly - bucket/*, here I was missing the * part
Endpoint in VPC must be created for glue to access S3 https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html
After these two settings, my glue job ran successfully. Hope this helps.
Make sure you have given the right policies.
I was facing the same issue, thought I had the role configured well.
But after I erased the role and followed this step, it worked ;]

Error with Data Pipeline backup when I transfer my data from DynamoDb to S3

I have to backup my DynamoDb table into S3 but when i launch this service I receive this error after three attempts:
private.com.amazonaws.AmazonServiceException: User:
arn:aws:sts::769870455028:assumed-role/DataPipelineDefaultResourceRole/i-3678d99c
is not authorized to perform: elasticmapreduce:ModifyInstanceGroups
(Service: AmazonElasticMapReduce; Status Code: 400; Error Code:
AccessDeniedException; Request ID:
9065ea77-0f95-11e5-8f35-39a70915a1ef) at
private.com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)
at
private.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)
at
private.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
at
private.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
at
private.com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.invoke(AmazonElasticMapReduceClient.java:1391)
at
private.com.amazonaws.services.elasticmapreduce.AmazonElasticMapReduceClient.modifyInstanceGroups(AmazonElasticMapReduceClient.java:785)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
private.com.amazonaws.services.datapipeline.retrier.RetryProxy.invokeInternal(RetryProxy.java:36)
at
private.com.amazonaws.services.datapipeline.retrier.RetryProxy.invoke(RetryProxy.java:48)
at com.sun.proxy.$Proxy33.modifyInstanceGroups(Unknown Source) at
amazonaws.datapipeline.cluster.EmrUtil.acquireCoreNodes(EmrUtil.java:325)
at
amazonaws.datapipeline.activity.AbstractClusterActivity.resizeIfRequired(AbstractClusterActivity.java:47)
at
amazonaws.datapipeline.activity.AbstractHiveActivity.runActivity(AbstractHiveActivity.java:113)
at
amazonaws.datapipeline.objects.AbstractActivity.run(AbstractActivity.java:16)
at
amazonaws.datapipeline.taskrunner.TaskPoller.executeRemoteRunner(TaskPoller.java:132)
at
amazonaws.datapipeline.taskrunner.TaskPoller.executeTask(TaskPoller.java:101)
at
amazonaws.datapipeline.taskrunner.TaskPoller$1.run(TaskPoller.java:77)
at
private.com.amazonaws.services.datapipeline.poller.PollWorker.executeWork(PollWorker.java:76)
at
private.com.amazonaws.services.datapipeline.poller.PollWorker.run(PollWorker.java:53)
at java.lang.Thread.run(Thread.java:745)
How can I do my backup? Does someone have this error?
thanks
edit: new policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:",
"dynamodb:",
"ec2:Describe*",
"elasticmapreduce:Describe*",
"elasticmapreduce:ListInstance*",
"elasticmapreduce:AddJobFlowSteps",
"elasticmapreduce:",
"rds:Describe",
"datapipeline:",
"cloudwatch:",
"redshift:DescribeClusters",
"redshift:DescribeClusterSecurityGroups",
"sdb:",
"sns:",
"sqs:"
],
"Resource": [
""
]
}
]
This is the new exception :
Error during job, obtaining debugging information... Examining task ID: task_1434014832347_0001_m_000008 (and more) from job job_1434014832347_0001 Examining task ID: task_1434014832347_0001_m_000013 (and more) from job job_1434014832347_0001 Examining task ID: task_1434014832347_0001_m_000005 (and more) from job job_1434014832347_0001 Examining task ID: task_1434014832347_0001_m_000034 (and more) from job job_1434014832347_0001 Examining task ID: task_1434014832347_0001_m_000044 (and more) from job job_1434014832347_0001 Examining task ID: task_1434014832347_0001_m_000004 (and more) from job job_1434014832347_0001 Task with the most failures(4): ----- Task ID: task_1434014832347_0001_m_000002 URL: http://ip-10-37-138-149.eu-west-1.compute.internal:9026/taskdetails.jsp?jobid=job_1434014832347_0001&tipid=task_1434014832347_0001_m_000002 ----- Diagnostic Messages for this Task: Error: Java heap space FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs
Datapipeline agent (TaskRunner) running on your EMR cluster is trying to resize the EMR cluster and it is failing. Your resource role that you passed to EMR cluster does not have permissions to invoke the following api AmazonElasticMapReduce::modifyInstanceGroups.
I just looked at the DefaultResourceRolePolicy, which is created using a wizard in console, (http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-iam-roles.html )
These are the allowed policies for emr:
"elasticmapreduce:Describe*",
"elasticmapreduce:ListInstance*",
"elasticmapreduce:AddJobFlowSteps"
and i found that it does not allow ModifyInstanceGroups.
Please update your resource role policy to allow that. E.g.,"elasticmapreduce:*"
Thx for reporting this bug. In the meanwhile, we will work on fixing the console wizard generated default resource role policy.
Aravind R.