I've read https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-custom-ami.html on how to create a custom AMI for use with AWS Data Pipeline. The document mostly makes sense except for the last line:
"Create and configure a user account named ec2-user."
What does this mean? What credentials (if any?) do I use for this AMI user? What SSH key do I allow to log in here?
The keypair that you should use can be specified in EC2 resource definition. See https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-resources-vpc.html
In case of the above EC2 resource snippet, my-key-pair would have a PEM file associated with it. Your ec2-account should correspond to that PEM file so that DPL can login to that EC2 resource.
Related
I am trying to create a terraform script which is creating AWS IAM user with secret key and access key id. Same script is also creating AWS EC2.
now where I am getting stuck is in to set those keys in ~/.aws/credential file(like aws configure does) in EC2.
What you're trying to do is not the recommended solution. If you want to grant any access to your EC2 instances then do so via IAM Roles rather than keys. You have to attach a Role with permissions to your EC2 and it can call other services same like it would do from AWS profile (using keys).
Reference to how to achieve this in Terraform: https://skundunotes.com/2021/11/16/attach-iam-role-to-aws-ec2-instance-using-terraform/
I have EC2 instance.
I'm trying to call aws s3 from it but getting an error
Unable to locate credentials
I tried aws configure which does show everything as empty.
I see IAM role for S3 full permissions assigned to this instance.
Do I need any additional configuration?
If you run aws on an Amazon EC2 instance that has an assigned role, then it should find the credentials automatically.
You can also use this to view the credentials from the instance:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
The role name should be listed. Then append it to the command:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME/
You should then be presented with AccessKeyId, SecretAccessKey, etc.
If that information does not appear, it suggests that your role is not correctly attached to the instance. Try unassigning the role and then assign it to the instance again.
First read and follow John Rotenstein's advice.
The SDK (from which the CLI is built from) searches for credentials in the following order:
Environment variables (AWS_ACCESS_KEY_ID ....)
Default Profile (credentials file)
ECS Container Credentials (if running on ECS)
Instance Profile
After verifying that your EC2 instance has credentials in the metadata, double check 1 & 2 to make sure that there are no other credentials present even if empty.
Note: This link argues with my last point (empty credentials file). I never create (store) credentials on an EC2 instance and I only use IAM Roles.
Instance Metadata
As far as I can tell the only way to mount an s3 bucket with s3fs is to use an accesskey:secretkey specified in a file with various file locations supported.
However, if I'm an ec2 instance, in the local s3 account, with an instance profile, I just want to use the instance profile credentials that are available. Does anyone know of a way to use an instance profile, and not have to set credentials in the local file system? If not, is anyone working on supporting this feature going forward?
Thanks
Once/if you have a role that is attached to the EC2 instance, you can then add the following entry in /etc/fstab to automatically mount the S3 bucket on boot:
s3fs#bucketname /PATHtoLocalMount fuse _netdev,iam_role=nameofiamrolenoquotes
Naturally, you have to have s3fs installed (as you do judging from the question), and the role policy must grant the appropriate (probably full) access to the S3 bucket. This is great in the sense that no IAM credentials need to be stored on the instance (=safer, because the role access cannot be used outside the instance attached to the role, while the IAM credentials can).
I am mounting an AWS S3 bucket as a filesystem using s3fs-fuse. It requires a file which contains AWS Access Key Id and AWS Secret Access Key.
How do I avoid the access using this file? And instead use AWS IAM roles?
As per Fuse Over Amazon document, you can specify the credentials using 4 methods. If you don't want to use a file, then you can set AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables.
Also, if your goal is to use AWS IAM instance profile, then you need to run your s3fs-fuse from an EC2 instance. In that case, you don't have to set these credential files/environment variables. This is because while creating the instance, if you attach the instance role and policy, the EC2 instance will get the credentials at boot time. Please see the section 'Using Instance Profiles' in page 190 of AWS IAM User Guide
there is an argument -o iam_role=--- which helps you to avoid AccessKey and SecretAccessKey
The Full steps to configure this is given below
https://www.nxtcloud.io/mount-s3-bucket-on-ec2-using-s3fs-and-iam-role/
In order to run AWS monitoring scripts (http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts-perl.html) on one of my ec2 existing instance, I need a IAM role associated to the instance.
I found a lot of documentation to launch an instance with a role... But nothing to link a role to an existing instance.
Is it possible?
If not, what can I do? Launch a new instance with a role and transfert the volumes to it?
If not, what can I do? Launch a new instance with a role and transfert
the volumes to it?
This is one option but it might take you a long time if you have a lot instances.
The other option is to simply use IAM by creating a new user then add the Amazon CloudWatch PutMetricData operation permission to that user. Then, create AWS credentials for that user and finally use them as per the docs that you specified:
Optional: If you aren't using an IAM role, update the
awscreds.template file that you downloaded earlier. The content of
this file should use the following format:
AWSAccessKeyId=YourAccessKeyID
AWSSecretKey=YourSecretAccessKey
Also set the environment variable AWS_CREDENTIAL_FILE to point that awscreds.template in the environment of the user that is running the mon-scripts-perl
There is a new method available associate-iam-instance-profile to Associates an IAM instance profile with a running or stopped instance.
Example:
aws ec2 associate-iam-instance-profile --instance-id YourInstanceId --iam-instance-profile Name=YourNewRole-Instance-Profile
Doco
AWS doesn't allow you to modify the instance role after launching the instance.
You can either:
1) Launch a new instance with the role needed by taking the AMI of the already running instance and reassigning the EIP.
or
2) Create a headless user, generate access key and secret key for the user with specific permssion, and use those keys.
Earlier there was no solution. You had to create and AMI and launch the server again with the appropriate role.
On Feb 9, AWS launched these new CLI options which can help you solve your problem.
See this link: https://aws.amazon.com/blogs/security/new-attach-an-aws-iam-role-to-an-existing-amazon-ec2-instance-by-using-the-aws-cli/