I'm trying to create a cluster using aws cli emr command. However, I can't seem to be able to create-default-roles needed before calling aws emr create-cluster
$ aws emr create-default-roles
A client error (NoSuchEntity) occurred when calling the GetRole operation: Unknown
I have made sure that my user has the following permissions:
IAMFullAccess - AWS Managed policy
AmazonElasticMapReduceforEC2Role - AWS Managed policy
AmazonElasticMapReduceFullAccess - AWS Managed policy
Any tips? Is there a place where I can just copy the roles json and create them manually?
The reason I started to do this is because when I run aws emr create-cluster it returns a cluster-id. But when that cluster-id is queries it state is set to terminated with the error: EMR service role arn:aws:iam::141703095098:role/EMR_DefaultRole is invalid
I DID manage to add these roles using the console by going to:
My Security Credentials > Roles > Create New Role
First Role with the following properties:
name: EMR_DefaultRole
policy: AmazonElasticMapReduceRole
Second Role with the following properties:
name: EMR_EC2_DefaultRole
policy: AmazonElasticMapReduceforEC2Role
Unfortunately I didn't get the command-line to work, but I suspect I might be something to do with my local setup.
I had issues with the console. With the client this worked:
# upgrade aws cli (can't hurt)
pip install --upgrade --user awscli
# aws configure process if you haven't (look it up)
# delete all the defunct shizzles
aws iam remove-role-from-instance-profile --instance-profile-name EMR_EC2_DefaultRole \
--role-name EMR_EC2_DefaultRole
aws iam delete-instance-profile \
--instance-profile-name EMR_EC2_DefaultRole
aws iam detach-role-policy \
--role-name EMR_EC2_DefaultRole \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role
aws iam delete-role --role-name EMR_EC2_DefaultRole
aws iam detach-role-policy --role-name EMR_DefaultRole \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole
aws iam delete-role --role-name EMR_DefaultRole
# now re-create them
aws emr create-default-roles
Note if you have attached policies, you might have to go into the console and delete them or find the appropriate aws cli command.
Source (our product is buggy and our role system is cumbersome, but if you buy premium support we'll tell you the workarounds):
https://aws.amazon.com/premiumsupport/knowledge-center/emr-default-role-invalid/
Related
I am trying to setup an ec2 instance (A role is associate with this instance).
This instance is responsible to
Create build, and upload to s3 bucket
Create a new application version from this build for elasticbeanstalk
Deploy newly created version on beanstalk
I am running following 3 commands. first 2 are executed successfully.
aws s3 cp api-service-build.zip s3://build-bucket/api-service/2022-11-2022.zip
aws elasticbeanstalk create-application-version
--application-name api-service-stage
--version-label v5
--description "Version 5"
--source-bundle S3Bucket="build-bucket",S3Key="api-service/2022-11-2022.zip"
but when I try to run third command its unable to deploy (please note on CLI its not failing)
aws elasticbeanstalk update-environment
--environment-name api-service-stage-env
--version-label v5
On beanstalk web console I can see following error
User: arn:aws:sts::xxxxxxxxx:assumed-role/MyAssumedRole/i-xxxxxx is not authorized to perform: autoscaling:DescribeAutoScalingGroups because no identity-based policy allows the autoscaling:DescribeAutoScalingGroups action (Service: AmazonAutoScaling; Status Code: 403; Error Code: AccessDenied;
I have updated my policy more than 30 times, to reach to above point, and yet another permission error.
Is there a way or a tool, where I paste my command and it tells me what permissions are required to run this command
aws s3 cp
aws elasticbeanstalk create-application-version
aws elasticbeanstalk update-environment
Permission I have added so far in MyAssumedRoles are as follows, I have added these with lots of hit and tries and yet its asking about another one autoscaling.
S3 Full access
Elastic Beanstalk full access
CloudFormation full access
Based on the error you are missing the AutoScaling permissions. They are different from the ones that you have already added. The best way to test is to use the AWS Policy Simulator. Follow the steps below :
Login to the AWS Console.
Go to the following URL : https://policysim.aws.amazon.com
Under User, Groups & Roles : Select Roles and then the role : MyAssumedRole
You can test the access on the right by selecting the action e.g. under Policy Simulator, select Auto Scaling and then action - DescribeScalingPlans. Policy Simulator will give you the exact policy you need to add for your role.
We're having an issue where we're on a CentOS EC2 instance that is using a role through an attached instance profile. When we're on the console after SSHing in, we run the python awscli command line tool to get our identity:
$ aws sts get-caller-identity
we're getting
An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid
other commands, such as aws ec2 describe-instances work and are allowed by the instance profile.
From reading the AWS documentation, no permissions should be required to get-caller-identity and there's no explicit deny set on the role associated with instance.
We checked and there's no .aws/credentials file and no env variables set, so access should be entirely managed through the metadata service on the EC2 instance.
Is there something missing in our setup or invocation of the awscli that might cause the permission to fail?
Just documenting the fix for anyone that runs into this issue.
All calls to the awscli should probably include a --region <region> parameter.
E.g.
$ aws sts get-caller-identity --region us-east-2
We were prompted for the region on our aws ec2 describe-instances call but on the aws sts get-caller-identity call, it just failed.
Additionally, we found that the AWS_REGION environment variable didn't seem to affect calls: we still needed to include the --region <region> parameter.
How do we update an already existing iam policy with a new json file using aws cli commands?
I already have a policy named mypolicy. I want to update this policy with a new json file (which has access to different resources). How do I perform this using aws cli command?
I tried to find the cli command. I found the command below from an AWS page:
aws organizations update-policy \
--policy-id p-examplepolicyid111 \
--content "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"*\"}}"
However, with this command I did not find any way to replace the policy with new json.
create policy version.
aws iam create-policy-version
--policy-arn arn:aws:iam::123456789012:policy/MyPolicy
--policy-document file://NewPolicyVersion.json --set-as-default
https://docs.aws.amazon.com/cli/latest/reference/iam/create-policy-version.html
list policies managed by customer
aws iam list-policies --scope Local
get a policy by arn
aws iam get-policy --policy-arn arn:aws:iam::123456789012:policy/MyPolicy
I want to create a lamba deployment package in python (with dependencies) using the Amazon tutorial.
When I push the .zip package with
aws lambda update-function-code --function-name my-function --zip-file fileb://function.zip
I get the following error
An error occurred (AccessDeniedException) when calling the UpdateFunctionCode operation:
User: arn:aws:iam::<ACCOUNT-ID>:user/jeanclaude is not authorized to perform: lambda:UpdateFunctionCode
on resource: arn:aws:lambda:eu-west-3:<ACCOUNT-ID>:function:my-function
Which policy should I grant to jeanclaude to give him the correct access?
The User created in AWS IAM which is configured with your AWS CLI using access_key and secret_key should have enough privileges to interact with AWS Lambda.
I would prefer AWSLambdaFullAccess policy attached to your User/Role. This is just for testing purpose and later you can reduce the privileges if you want.
Once you have done the above then if you run the command
aws lambda update-function-code --function-name "helloworld" --zip-file "fileb://./helloworld.zip" --region "eu-west-2"
it should work, note that for update-function-code mandatory field is just the --function-name other fields are optional.aws cli update-fuction-code
Also please take a note of the create-function command it has just the following fields as mandatory and all other are optional aws cli docs
create-function
--function-name <value>
--runtime <value>
--role <value>
--handler <value>
and the --role here is the role required by the lambda while executing to interact with other services (not to be confused by the user above)
The user needs permission to UpdateFunctionCode for that ARN. More specific information is here.
I have an ec2 instance which has the IAM role attached to it for creation of EC2 instances. It also has aws cli installed.
I am trying to use cloudformation cli command to create a stack based on a json template in s3. But it's asking me to provide accessKeyId and secret key etc which I do not have as I do not want to create any IAM user.
Is there any workaround to this ?
Here is the command which I am using:
aws --region us-east-1 cloudformation create-stack --stack-name myteststack --template-url https://s3.console.aws.amazon.com/s3/object/aws-cli-test11/test_ct.json?region=us-east-1&tab=properties --parameters ParameterKey=KeyName,ParameterValue=eai_ec2user,ParameterKey=InstanceType,ParameterValue=t2.medium
EDIT----------
I receive the error messaage Unable to locate credentials. You can configure credentials by running "aws configure". on all aws cli commands. I tried below command and got the same error.:
aws ec2 describe-instances --filters "Name=availability-zone,Values=us-east-1a"
The question is why it's not using the IAM role which is assigned to the instance. That role is having all the necessary policies attached.
The commands are working fine under root user but not under non root user. All the above errors are for non root users.