I am using AWS CLI to create AWS spot instance but each time I am getting the below error.
"An error occurred (UnauthorizedOperation) when calling the RequestSpotInstances operation: You are not authorized to perform this operation."
Note :- I am able to create spot instance from EC2 console(GUI) successfully.
below mention is aws cli
aws ec2 request-spot-instances --spot-price "0.003" --instance-count 1 --type "persistent" --launch-specification file://param.json --instance-interruption-behavior "stop" --profile ""
content for param.json
{
"ImageId": "ami-0123456",
"UserData":"file:://my_script.txt"
"KeyName": "gistdev_default_rsa",
"SecurityGroupIds": [ "sg-0123456" ],
"InstanceType": "t2.medium",
"Placement": {
"AvailabilityZone": "us-west-2a"
},
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "subnet-0123456",
"Groups": [ "sg-0123456" ],
"AssociatePublicIpAddress": true
}
]
}
I have not specified "IamInstanceProfile" parameter in json file. Is this mandatory to have it?
~Ashish
Just because you can create spot instances using console (GUI) does not mean you can create them using CLI. What can go wrong?
Is API/SDK access enabled for the user?
How is the credentials supplied to the CLI? using access/secret or using metadata server (IAMProfile)
Is the credentials supplied to the CLI different from the one for the IAM user using the console/GUI? (most likely reason)
Is the CLI getting credentials in some other way? See: Configuration Settings and Precedence
There is missing permission to iam user of AmazonEC2SpotFleetRole .Try after attaching AWS managed AmazonEC2SpotFleetRole policy to you iam user.
There is no mandatory of add "IamInstanceProfile" in json file while creating spot-instance . IamInstanceProfile is a Role that can be accessed via ec2 entity as a trusted enitity .
When you create a new user, you have the option to grant them "Programmatic access" and/or "AWS Managed Console access"
You need to be sure "Programmatic access" is enabled for users who need to use the CLI.
Related
I've currently writing a Terraform module for EC2 ASGs with ECS. Everything about starting the instances works, including IAM-dependent actions such as KMS-encrypted volumes. However, the ECS agent fails with:
Error getting ECS instance credentials from default chain: NoCredentialProviders: no valid providers in chain.
Unfortunately, most posts I find about this are about the CLI being run without credentials configured, however this should of course use the instance profile.
The posts I do find regarding that are about missing IAM policies for the instance role. However, I do have this trust relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com",
"ecs.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
(Added ECS because someone on SO had it in there, I don't think that's right. Also removed some conditions.)
It has these policies attached:
AmazonSSMManagedInstanceCore
AmazonEC2ContainerServiceforEC2Role
When I connect via SSH and try to run awscli commands, I get the error
Unable to locate credentials. You can configure credentials by running "aws configure".
running any command.
But with
curl http://169.254.169.254/latest/meta-data/iam/info
I see the correct instance profile ARN and with
curl http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
I see temporary credentials. With these in the awscli configuration,
aws sts get-caller-identity
returns the correct results. There are no iptables rules or routes blocking access to the metadata service and I've deactivated IMDSv2 token requirement for the time being.
I'm using the latest stable version of ECS-optimized Amazon Linux 2.
What might be the issue here?
When trying to create an apprunner service using aws apprunner create-service --cli-input-json file://./myconfig.json, I get the error in title:
An error occurred (InvalidRequestException) when calling the CreateService operation: Error in assuming access role arn:aws:iam::1234:role/my-role
The myconfig.json I'm using is fairly similar to example json from AWS CreateService docs, & I don't think it's particularly relevant here.
The error seems to imply I should assume the role... but I've already assumed the role with this command from this stackoverflow q/a:
eval $(aws sts assume-role --role-arn arn:aws:iam::1234:role/my-role --role-session-name apprunner-stuff1 --region us-east-1 | jq -r '.Credentials | "export AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=\(.SessionToken)\n"')
This runs without error & when I run:
aws sts get-caller-identity
it outputs the following which looks correct I think:
{
"UserId": "SOME1234NPC:apprunner-stuff1",
"Account": "1234",
"Arn": "arn:aws:sts::1234:assumed-role/my-role/apprunner-stuff1"
}
At this point, the error message doesn't make sense & I'm wondering what dumb IAM thing am I doing wrong?
Apprunner specific wise - I've attempted to to give my-role all the permissions from AppRunner IAM doc to run CreateService, but I could easily have missed some. The error message here doesn't seem to indicate that the role doesn't have sufficient permissions, but might be relevant.
Instead of trying to create a role following IAM doc permissions, I followed the UI AppRunner guide here. That created a role that was auto named AppRunnerECRAccessRole. I used that role as my AccessRoleArn in the json configuration, making that json config section look like:
"AuthenticationConfiguration": {
"AccessRoleArn": "arn:aws:iam::12345:role/service-role/AppRunnerECRAccessRole"
},
I followed this stackoverflow q/a to allow my user / group to assume the AppRunnerECRAccessRole, with a policy applied to the user/group like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::12345:role/my-role",
"arn:aws:iam::12345:role/service-role/AppRunnerECRAccessRole"
]
}
]
}
After this I was just able to run:
aws apprunner create-service --cli-input-json file://./myconfig-with-ui-role-arn.json
& it worked! (without even assuming the role via eval command). Though I gave the user access to both roles, creating only worked via the new AppRunnerECRAccessRole role. So I think the takeaway / main answer is to create an AppRunner service via UI & then reuse its service role.
I launched an ec2 instance and created a role with a full S3 access policy for the instance. I installed awscli on it and configured my user's access key. My user has admin access and full S3 access policy too. I can see the buckets in the aws console but when I try to run aws s3 ls on the instance it returned An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied.
What else I need to do to add permission to the role or my user properly to be able to list and sync object between S3 and the instance?
I ran into this issue as well.
I ran aws sts get-caller-identity and noticed that the ARN did not match what I was expecting. It turns out if you have AWS configurations set in your bash_profile or bashrc, the awscli will default to using these instead.
I changed the enviornment variables in bash_profile and bashrc to the proper keys and everything started working.
Turns out I forgot I had to do mfa to get access token to be able to operate in S3. Thank you for everyone response.
There appears to be confusion about when to use IAM Users and IAM Roles.
When using an Amazon EC2 instance, the best method to grant permissions is:
Create an IAM Role and attach policies to grant the desired permissions
Associate the IAM Role with the Amazon EC2 instance. This can be done at launch time, or afterwards (Actions/Instance Settings/Attach IAM Role).
Any application running on the EC2 instance (including the AWS CLI) will now automatically receive credentials. Do not run aws configure.
If you are wanting to grant permissions to your own (non-EC2) computer, then:
Create an IAM User (or use your existing one) and attach policies to grant the desired permissions
On the computer, run aws configure and enter the Access Key and Secret Key associated with the IAM User. This will store the credentials in ~/.aws/credentials.
Any application running on this computer will then use credentials from the local credentials file
Create a IAM user with permission.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": "arn:aws:s3:::bucketName/*"
}
]
}
Save Access key ID & Secret access key.
sudo apt install awscli
aws configure
AWS Access Key ID [None]: AKIAxxxxxxxxxxxZI4
AWS Secret Access Key [None]: 8Bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8
Default region name [None]: region (ex. us-east-2)
Default output format [None]: json
aws s3 ls s3://s3testingankit1/
This problem can occurs not only from the CLI but also when executing S3 API for example.
The reason for this error can come from wrong configuration of the access permissions to the bucket.
For example with the setup below you're giving a full privileges to perform actions on the bucket's internal objects, BUT not specifying any action on the bucket itself:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::<name-of-bucket>/*"
]
}
]
}
This will lead to the mentioned
... (AccessDenied) when calling the ListBuckets ...
error.
In order to fix this you should allow application to access the bucket (1st statement item) and to edit all objects inside the bucket (2nd statement item):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<name-of-bucket>"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::<name-of-bucket>/*"
]
}
]
}
There are shorter configurations that might solve the problem, but the one specified above tries also to keep fine grained security permissions.
I ran into this yesterday running a script I ran successfully in September 2021.
TL;DR: add --profile your.profile.name to the end of the command
I have multiple profiles on the login I was using. I think something in the aws environment changed, or perhaps I had done something that was able to bypass this before. Back in September I set the profile with
aws configure set region us-west-2 --profile my.profile.name
But yesterday, after the failure, I saw that aws sts get-caller-identity was returning a different identity. After some documentation search I found the additional method for specifying the profile, and operations like:
aws s3 cp myfile s3://my-s3-bucket --profile my.profile.name
all worked
I have an Windows machine with CyberDuck from which I was able to access a destination bucket, but when trying to access the bucket from a Linux machine with aws command, I got "An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied".
I then executed same command "aws s3 ls" from a command line interface on the Windows machine and it worked just fine. It looks like there is some security restriction on the AWS server for the machine/IP.
I am starting migrating our secrets from AWS parameter store to AWS secret manager, currently I am facing a problem that I don't how to solve, could anyone provide any insight?
We have a AWS account (let's call it identity account), which we manage all IAM users and groups. And we have another AWS account which hosts our infra (let's call this infra account). We'd like to manage all users in the identity account and let user assumes to poweruser role in the infra account so we can manage all users at one place.
In the infra account, we have RDS running and we want to create DB users for our developers so they can login to database for debugging purpose, but we also want to audit what they have done in case someone did something bad to our database, so we need to create one DB user per developer. All of those DB credentials are saved into AWS secret with a naming convention like
/dev/rds/mysql/users/foo
/dev/rds/mysql/users/bar
So here is the question: how can I manage user's IAM policy to restrict the permission of user so they can ONLY access their own secret? From this AWS doc that we CANNOT get aws:username when user is using assumed role to access AWS, thus the following policy would never work
actions = [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
]
resources = [
"arn:aws:secretsmanager:us-east-1:12345678912:secret:/dev/rds/mysql/users/${aws:username}"
]
The only IAM variable that I can use for assumed role is aws:userid but it would be something like this (assume user foo's username in identity account is foo#emaildomain.com)
"AROAJGHLP6KERYI375PJY:foo#emaildomain.com"
It also looks like that the role-id (AROAJGHLP6KERYI375PJY in this example) is random, with prefix AROA, which means I CANNOT use following policy either (and plus, having AROAJGHLP6KERYI375PJY:foo#emaildomain.com as a secret name in secret manager is pretty ugly)
actions = [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
]
resources = [
"arn:aws:secretsmanager:us-east-1:12345678912:secret:/dev/rds/mysql/users/${aws:userid}"
]
Currently my policy ended up with this
actions = [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
]
resources = [
"arn:aws:secretsmanager:us-east-1:12345678912:secret:/dev/rds/mysql/users/*"
]
which means as long as user assumed to infra account, they have access to other developers DB credentials as well.
I've looked into CloudWatch metric to see if I can setup a filter to filter out the API call that user foo is calling GetSecretValue API to get user bar's credential, but CloudWatch filter doesn't support user REGEX to extract certain value from JSON. Here's the example of the GetSecretValue event from CloudTrail log:
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAJGHLP6KERYI375PJY:foo#emaildomain.com",
"arn": "arn:aws:sts::12345678912:assumed-role/poweruser/foo#emaildomain.com",
"accountId": "12345678912",
"accessKeyId": "ASIATA5XIF7AFC2CQ7NO",
"sessionContext": {
"attributes": {
"mfaAuthenticated": "true",
"creationDate": "2018-07-11T21:31:20Z"
},
"sessionIssuer": {
"type": "Role",
"principalId": "AROAJGHLP6KERYI375PJY",
"arn": "arn:aws:iam::12345678912:role/poweruser",
"accountId": "12345678912",
"userName": "poweruser"
}
}
},
"eventTime": "2018-07-11T21:32:56Z",
"eventSource": "secretsmanager.amazonaws.com",
"eventName": "GetSecretValue",
"awsRegion": "us-east-2",
"sourceIPAddress": "1.2.3.4",
"userAgent": "aws-internal/3",
"requestParameters": {
"secretId": "/dev/rds/mysql/users/foo"
},
"responseElements": null,
"requestID": "f98ad2c2-8551-11e8-8a3f-751b0a8a6ca5",
"eventID": "73b8de89-bc8c-41a3-a172-58dd8d79a026",
"eventType": "AwsApiCall",
"recipientAccountId": "12345678912"
}
If I can extract foo#emaildomain.com from { $.userIdentity.principalId } and extract foo from to { $.requestParameters} then I can try some magic to compare foo#emaildomain.com == foo to trigger alert if user is trying to get other people's credential, but, I can't...
So, in this case, how could I manage my policy to lock users' permission?
From your first example, for each user, can you try to hardcode the secret in their user iam policy?
So for user foo, the user policy would look something like
...
actions = [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue",
"secretsmanager:UpdateSecretVersionStage"
]
resources = [
"arn:aws:secretsmanager:us-east-1:12345678912:secret:/dev/rds/mysql/users/foo"
]
...
You would have the same structure for bar, ...
If you have many users, this solution wouldn't scale
The problem seems to be that you are using a common assumed role for all the users. An alternative to this would be to create a resource policy on each secret that granted access permissions to the owning user in the identity account. This would let the users access the secret directly without calling assumerole. This would not prevent them from still assuming the infra account poweruser role and accessing the secret, so you would either have to drop Secrets Manager privileges from the role, or explicitly deny the infra power user in the resource policy you add to the secret.
Setting up cross account access like this also means you can not use the default KMS encryption key. You will need to setup a custom KMS key that grants the correct access permission to the identity account and re-encrypt the secrets with that new key.
Since resource policies and custom KMS keys can not be setup in the Secrets Manager Console, this all requires using the CLI or one of the SDKs.
I have a custom docker image uploaded to ECS. I opened up the permissions to try and get through this issue (I will lock it down again once I can get this to work). I am attempting to deploy the docker image to elastic beanstalk. I have a docker enabled elastic beanstalk environment set up. According to the AWS docs, if I am pulling my image from within AWS, I don't need to pass in credentials. So I upload my Dockerrun.aws.json file and attempt to install it. It fails with the error:
Command failed on instance. Return code: 1 Output: Failed to authenticate with ECR for registry '434875166128' in 'us-east-1'. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03build.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
The /var/log/eb-activity.log information has nothing useful in it.
Here's my Dockerrun.aws.json file:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "{id000xxxx}.dkr.ecr.us-east-1.amazonaws.com/my-repo:1.0.0",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "4000"
}
],
"Logging": "/var/log/app-name"
}
I have also tried adding the authentication with the dockercfg.json file in S3. It didn't work for me either.
Note that I am using a business account instead of a personal account, so there may be some unknown variances as well.
Thanks!
Update: My user has full permissions at the moment too, so there shouldn't be anything permission-wise getting in the way.
I was having the same problem.
Solution:
In AWS -> IAM -> Roles - > pick the role your beanstalk is using.
In my case it was set to aws-elasticbeanstalk-ec2-role
Under Permissions for the role, attach policy: AmazonEC2ContainerRegistryReadOnly
In ECR there is no need to give any permissions to this role.
Assuming
You are using Terraform to provision your infrastructure
You have created a sample ElasticBeanstalk app at least once, so that you have the default role created.
The default ElasticBeanstalk role is named: aws-elasticbeanstalk-ec2-role
Then you can comfortably use the following format to add ECR Read Only policy to the role:
data "aws_iam_role" "elastic_beanstalk_role" {
name = "aws-elasticbeanstalk-ec2-role"
}
resource "aws_iam_policy" "ebs_ecr_policy" {
name = "aws-elasticbeanstalk-ec2-ecr-policy"
description = "Enable elastic-beanstalk to be able to access ECR repository with images"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:GetLifecyclePolicy",
"ecr:GetLifecyclePolicyPreview",
"ecr:ListTagsForResource",
"ecr:DescribeImageScanFindings"
],
"Resource": "*"
}
]
}
EOF
}
resource "aws_iam_policy_attachment" "ebs_ecr-policy-attach" {
name = "ebs-ecr-policy-attachment"
roles = [data.aws_iam_role.elastic_beanstalk_role.name]
policy_arn = aws_iam_policy.ebs_ecr_policy.arn
}
This way you can manage updates to the role and policy from your infrastructure code.
You can intialize necessary service roles for elastic beanstalk (aws-elasticbeanstalk-ec2-role , aws-elasticbeanstalk-service-role , AWSServiceRoleForECS ) by using the new console of Elastic Beanstalk.
You have to do this only one time on each AWS account :
Go to the Elastic beanstalk console.
Accept the "new design" : in the top of the console, if see a message "we re testing a new design", optin to accept to use the new version of the console. Warning, it seems you cant rollback to the old console.
Start the Create New Application wizard, and use a default sample application in the technology.
Complete all the step of the wizard until the resume, and look at the Security pannel : you will see the two roles "aws-elasticbeanstalk-ec2-role" and "aws-elasticbeanstalk-service-role". And terminate the wizard to create the sample app.
After a while, the application should be running
In case of emergency, go to the IAM console and delete the roles aws-elasticbeanstalk-ec2-role and aws-elasticbeanstalk-service-role and run the wizard again.
I fixed the "Command failed on instance. Return code: 1 Output: Failed to authenticate with ECR for registry" and an other strange error ("The AWS Access Key Id you provided does not exist in our records. (ElasticBeanstalk::ManifestDownloadError)") by using the NEW console. I still had this error with the old one.