I have a CloudFormation based deployment of an application, which creates an Amazon Linux 2 EC2 instance from a Marketplace AMI with an IAM Instance Profile. In the CloudFormation template, I am running scripts via cfn-init which have some AWS CLI commands in them, like ssm get-parameter, and also am mounting an EFS volume using mount and amazon-efs-utils. This has been working like a charm.
Now I have a customer who is running in their AWS account using the common AMI and CloudFormation templates. For them, the cfn-init is failing because cfn-init is running as root, but root strangely has no IAM privileges and can't run the scripts or the efs-helper based mount, even though the Instance Profile is there. But ec2-user does have IAM privileges from the Instance Profile!
To summarize:
ec2-user does have IAM privileges from the Instance Profile.
# logged in as ec2-user
aws ssm get-parameter --name "/aParameter"
returns a result
root does not have IAM privileges from the Instance Profile.
# logged in as ec2-user
sudo aws ssm get-parameter --name "/aParameter"
An error occurred (ParameterNotFound) when calling the GetParameter operation:
I expected that all users running in the instance would have the Instance Profile as their credentials if they have not explicitly authenticated some other way. I can't see anything in the customer's environment that would cause this - I was thinking maybe a Service Control Policy - but they have none of that.
Has anyone seen this behavior and have a fix?
Thanks so much in advance.....
I have no idea about the relationship between ec2-user and instance profile.
ec2-user is a user on linux.
instance profile is an IAM role attached to an EC2 instance.
I think you can check with other commands like aws s3 ls.
Btw, you are using sudo with root.
Turns out AWS credentials were left in the root user on the AMI. :-/
Asri Badlah's aws iam get-user suggestion showed the problem. Thank you!
The instance profile did not have access to get-user, but sudo get-user did and showed the user from the credentials in /root/.aws.
Related
Background:
I have jenkins installed in AWS Account #1 (account1234) and it has iam Role-jenkins attached to it. There's github configured with Jenkins.
When I click build job in Jenkins, jenkins pulls all the files from github and can be found in
/var/lib/jenkins/workspace/.
There's an application running in AWS Account #2 (acccount5678) in an ec2 instance (i-xyz123) and the project files are in /home/app/all_files/ ; This ec2 instance role has app-role attached to it.
What I'm trying to achieve:
When I click build, I want jenkins to push files from account 1234 to account 5678 by opening an SSM session from Jenkins ,to the ec2 instance on which app is running.
What I tried:
In the jenkins as part of build shell script I added:
aws ssm send-command --region us-east-1 --instance-ids i-xyz123 --document-name AWS-RunShellScript --comment IP config --parameters commands=ifconfig --output text
to test it. (If successful, I want to pass cp var/lib/jenkins/workspace/ /home/app/all_files/ as the command)
Error:
An error occurred (AccessDeniedException) when calling the SendCommand operation: User: arn:aws:sts::account1234:assumed-role/Role-Jenkins/i-01234abcd is not authorized to perform: ssm:SendCommand on resource: arn:aws:ec2:us-east-1:account1234:instance/i-xyz123
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Issue 1: instance/i-xyz123 is in account5678 but error above shows ssm trying to connect to instance in account1234 ( which shouldn't be happening)
Q1: How do I update my command so that it tries to open an ssm session
with instance/i-xyz123 present in account5678 to accomplish what I'm
trying to do.
I believe I would also need to make each role added as a trusted relationship to the other.
(Note I want to do it via sessions manager as I won't have to deal with credentials of any sort)
If I've understood correctly then you're right; to interact with the resources in account5678, there needs to be a trust relationship so that the Jenkins account can assume the relevant role in account5678 and call SSM from there.
Once you've configured the role relationship (ref: IAM cross account roles )
You should be able to achieve what you need by assuming the role first in your shell script and then running the ssm command. That way Jenkins will use the temp creds and execute the command in the correct account (5678).
This site steps through it pretty well :
Tom Gregory - Jenkins Assume Role
If you just cmd/ctrl f on that page ^ and search for 'shell' you should get to the section you need. Hope this somewhat helps.
While debugging this question, I went on and
In IAM console at https://console.aws.amazon.com/iam/
1.1. Deleted one role (CodeDeployServiceRole).
1.2. Created a service role.
In S3 console at https://console.aws.amazon.com/s3/
2.1. Emptied and deleted one bucket (tiagocodedeploylightsailbucket).
2.2. Created a new bucket in EU London (eu-west-2).
Back into the IAM console at https://console.aws.amazon.com/iam/
3.1. Deleted one policy (CodeDeployS3BucketPolicy).
3.2. Created a new policy.
Stay in the IAM console at https://console.aws.amazon.com/iam/
4.1. Delete one user (LightSailCodeDeployUser)
4.2. Created a new user (with that same name).
Navigate to the Lightsail home page at https://lightsail.aws.amazon.com/
5.1. Deleted previous instance (codedeploy).
5.2. Created one new instance with Amazon Linux (Amazon_Linux_1) (Note that if I use Amazon Linux 2 then would reach this problem),
using the script
mkdir /etc/codedeploy-agent/
mkdir /etc/codedeploy-agent/conf
cat <<EOT >> /etc/codedeploy-agent/conf/codedeploy.onpremises.yml
---
aws_access_key_id: ACCESS_KEY
aws_secret_access_key: SECRET_KEY
iam_user_arn: arn:aws:iam::525221857828:user/LightSailCodeDeployUser
region: eu-west-2
EOT
wget https://aws-codedeploy-us-west-2.s3.us-west-2.amazonaws.com/latest/install
chmod +x ./install
sudo ./install auto
Checked that CodeDeploy agent is running and then when running the following command in AWS CLI
aws deploy register-on-premises-instance --instance-name Amazon_Linux_1 --iam-user-arn arn:aws:iam::525221857828:user/LightSailCodeDeployUser --region eu-west-2
I get
An error occurred (IamUserArnAlreadyRegisteredException) when calling
the RegisterOnPremisesInstance operation: The on-premises instance
could not be registered because the request included an IAM user ARN
that has already been used to register an instance. Include either a
different IAM user ARN or IAM session ARN in the request, and then try
again.
Even though I deleted the user, created one with the same name and then deleted the other existing instance, the IAM User ARN is still the same
arn:aws:iam::525221857828:user/LightSailCodeDeployUser
To fix it, I've gone back to step 4 and created a user with a different name; then, updated the script for the instance creation, checked if the CodeDeploy agent is running and now when running in AWS CLI
aws deploy register-on-premises-instance --instance-name Amazon_Linux_1 --iam-user-arn arn:aws:iam::525221857828:user/GeneralUser --region eu-west-2
I get the expected result
I am trying to get some files from S3 on startup in an EC2 instance by using a User Data script and the command
/usr/bin/aws s3 cp ...
The log tells me that permission was denied and I believe it is due to aws cli not finding any credentials when executing the user data script.
Running the command with sudo after the instance has started works fine.
I have run aws configure both with sudo and without.
I do not want to use cronjob to run something on startup since I am working with an AMI and often need to change the script, therefore it is more convenient for me to change the user data instead of creating a new AMI everytime the script changes.
If possible, I would also like to avoid writing the credentials into the script.
How can I configure awscli in such a way that the credentials are used when running a user data script?
I suggest you remove the AWS credentials from the instance/AMI. Your userdata script will be supplied with temporary credentials when needed by the AWS metadata server.
See: IAM Roles for Amazon EC2
Clear/delete AWS credentials configurations from your instance and create an AMI
Create a policy that has the minimum privileges to run your script
Create a IAM role and attach the policy you just created
Attach the IAM role when you launch the instance (very important)
Have your userdata script call /usr/bin/aws s3 cp ... without supplying credentials explicitly or using credentials file
You can configure your EC2 instance to receive a pre-defined IAM Role that has its credentials "baked-in" to the instance that it fetches upon instantiation, which it turn will allow it to call aws-cli commands in your User Data script without the need to configure credentials at all.
Here's more info on IAM Roles for EC2:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
It's worth noting here that you'll need to attach the appropriate policies to the IAM Role that you assign to your instance in order for the aws-cli commands to succeed. More information on that can be found here:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#working-with-iam-roles
I want to sync a s3 bucket on ec2 instance and I want to do it using ssm run-command. The problem is ssm run-command always executes as root user which does not have s3 permissions. how to run ssm run-commands as ubuntu user.
I have Ansible Master running on an ubuntu ec2 server with IAM role having full permission on Ec2 and nothing else. All the instances deployed using this Ansible-master are although deployed but in terminated state.
Albiet, while I was testing another approach, I created a new master and provided my authentication keys which are of a root user having all the permissions.
Is there a problem with IAM role's permissions or deployment is known not to work with IAM roles?
It works as expected for me:
root#test-node:~# cat /etc/issue
Ubuntu 14.04.4 LTS \n \l
root#test-node:~# ansible --version
ansible 2.1.2.0
config file =
configured module search path = Default w/o overrides
root#test-node:~# pip list | grep boto
boto (2.42.0)
If no credentials are specified in env variables or config files, Boto (library that Ansible uses to connect to AWS) will try to fetch credentials from instance metadata.
You may try to fetch them manually with:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>
and pass KeyId and Secret to Ansible via environment variables to test that role's permissions are correct.
Keep in mind, that:
IAM role should be attached to the EC2 instance before start
region should be always defined: either via module parameters or via environment variable.