In the logstash configuration file, I gave the following input plugin.
input{
s3{
bucket => 'bucket_name'
region => 'eu-west-1'
}
}
When I started logstash, it threw an error asking for AWS credentials. But I cannot provide AWS access_key_id and secret_key. I found that with IAM roles on EC2 instances, we shouldn't have to provide AWS credentials explicitly for an application that make those requests. I tried to understand how to configure IAM roles from a logstash Github issue, but failed. Please suggest how I should configure logstash file so that I can get data from S3 without providing AWS credentials explicitly.
When you are creating the EC2 instance, you may assign it an IAM role. If you don't give it a role when you create it, you cannot ever give it a role. You can modify the properties of a role (e.g. give the role more or less access) but you cannot change what role an instance has after the instance has been created.
The easiest way to test if an instance has the credentials you need is with the command-line tools:
$ echo $AWS_ACCESS_KEY_ID
$
That shows that we don't have the environment variable set, so if we're getting credentials, they must be instance creds.
$ aws --region=eu-west-1 s3 ls my.bucket
PRE some-directory/
PRE another-dir/
2015-11-05 18:03:53 464 little-file
2014-10-28 15:32:13 19740 bigger-file.html
$
I was able to list the bucket contents, so the EC2 instance where I'm running this command must have an IAM role that allows it to list this bucket contents!
Related
So I have created an IAM user and added a permission to access S3 then I have created an EC2 instance and SSH'ed into the it.
After giving "aws s3 ls" command, the reply was
"Unable to locate credentials. You can configure credentials by running "aws configure".
so what's the difference between giving IAM credentials(Key and Key ID) using "aws configure" and editing the bucket policy to allow s3 access to my instance's public IP.
Even after editing the bucket policy(JSON) to allow S3 access to my instance's public IP why am I not able to access the s3 bucket unless I use "aws configure"(Key and Key ID)?
Please help! Thanks.
Since you are using EC2 you should really use EC2 Instance Profiles instead of running aws configure and hard-coding credentials in the file system.
As for the your question of S3 bucket policies versus IAM roles, here is the official documentation on that. They are two separate tools you would use in securing your AWS account.
As for your specific command that failed, note that the AWS CLI tool will always try to look for credentials by default. If you want it to skip looking for credentials you can pass the --no-sign-request argument.
However, if you were just running aws s3 ls then that was trying to list all the buckets in your account, which you would have to have IAM credentials for. Individual bucket policies would not be taken into account in that scenario.
If you were running aws s3 ls s3://bucketname then that may have worked as aws s3 ls s3://bucketname --no-sign-request.
When you create iam user so there are two parts
policies
roles
Policies are attached to user, like what all services user can pr can't access
roles are attached to application, what all access that application can have
So you have to permit ec2 to access S3
There are two ways for that
aws configure
attach role to ec2 instance
while 1 is tricky and legthy , 2 is easy
Go to ec2-instance-> Actions -> Security -> Modify IAM role -> then select role (ec2+s3 access role)
thats it , you can simply do aws s3 ls from ec2 instance
Stupidly enough, I did delete by mistake my default AWS IAM user!
I used it for example do aws s3 sync...
Now the error I get is:
$ aws s3 sync build/ s3://mybucket.mydomain.com
fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.
Is there a way to recover?
I think I need instructions how to create a new user with the sufficient roles to enable my local aws cli to be able to do aws s3 sync ...
UPDATE: I did just create a new user on my AWS console, and added a policy (to start with) to list my bucket. The problem is I don't know how to attach my aws cli to that new user... :-(
If you are the only person using this AWS Account, then add the AdministratorAccess Policy to your IAM User. That will grant complete access.
Then, in the Security credentials tab of the IAM User click Create access key. Copy the Access Key and Secret Access Key.
On the command line, run aws configure and provide those keys to configure the user.
Test with: aws s3 ls
I have EC2 instance.
I'm trying to call aws s3 from it but getting an error
Unable to locate credentials
I tried aws configure which does show everything as empty.
I see IAM role for S3 full permissions assigned to this instance.
Do I need any additional configuration?
If you run aws on an Amazon EC2 instance that has an assigned role, then it should find the credentials automatically.
You can also use this to view the credentials from the instance:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
The role name should be listed. Then append it to the command:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME/
You should then be presented with AccessKeyId, SecretAccessKey, etc.
If that information does not appear, it suggests that your role is not correctly attached to the instance. Try unassigning the role and then assign it to the instance again.
First read and follow John Rotenstein's advice.
The SDK (from which the CLI is built from) searches for credentials in the following order:
Environment variables (AWS_ACCESS_KEY_ID ....)
Default Profile (credentials file)
ECS Container Credentials (if running on ECS)
Instance Profile
After verifying that your EC2 instance has credentials in the metadata, double check 1 & 2 to make sure that there are no other credentials present even if empty.
Note: This link argues with my last point (empty credentials file). I never create (store) credentials on an EC2 instance and I only use IAM Roles.
Instance Metadata
I am mounting an AWS S3 bucket as a filesystem using s3fs-fuse. It requires a file which contains AWS Access Key Id and AWS Secret Access Key.
How do I avoid the access using this file? And instead use AWS IAM roles?
As per Fuse Over Amazon document, you can specify the credentials using 4 methods. If you don't want to use a file, then you can set AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables.
Also, if your goal is to use AWS IAM instance profile, then you need to run your s3fs-fuse from an EC2 instance. In that case, you don't have to set these credential files/environment variables. This is because while creating the instance, if you attach the instance role and policy, the EC2 instance will get the credentials at boot time. Please see the section 'Using Instance Profiles' in page 190 of AWS IAM User Guide
there is an argument -o iam_role=--- which helps you to avoid AccessKey and SecretAccessKey
The Full steps to configure this is given below
https://www.nxtcloud.io/mount-s3-bucket-on-ec2-using-s3fs-and-iam-role/
I am trying to use aws cookbook with iam roles, but when I trying to not include aws_access_key and aws_secret_access_key in the aws_ebs_volume block, the chef keep showing an error: RightAws::AwsError: AWS access keys are required to operate on EC2.
I assume when cookbook mean omit the resource parameters aws_secret_access_key and aws_access_key, I just delete them from the block.
aws_ebs_volume "userhome_volume" do
provider "aws_ebs_volume"
volume_id node['myusers']['usershome_ebs_volid']
availability_zone node['myusers']['usershome_ebs_zone']
device node['myusers']['usershome_ebs_dev_id']
action :attach
end
Does anyone have the example of aws cookbook with iam roles please?
update:
Do I still need to define aws creeds data bag if I have already have proper iam role attached to the instance?
When I use iam role and aws cookbook, what does the was_ebs_volume block look like?
In order to manage AWS components, you need to provide authentication credentials to the nodein one of two ways:
explicitly pass credentials parameter to the resource
or let the resource pick up credentials from the IAM role assigned to the instance
When you provision the instance, you should assign it the appropriate role in "Step 3. Configure Instance Details" (when using the console). The setting "IAM role" for EC2 automatically deploys and rotates AWS credentials for you, eliminating the need to store your AWS access keys with your application. On an instance provisioned this way, you no longer need to include aws_access_key and aws_secret_access_key in the aws_ebs_volume block.
Here are code examples on how to launch an instance with an IAM role using the IAM and Amazon EC2 CLIs:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
and here are some code examples:
http://www.getchef.com/blog/2013/12/19/automating-iam-credentials-with-ruby-and-chef/
When you assign the appropriate IAM role during instance provisioning, your code should work without aws_access_key and aws_secret_access_key.
Here are the steps:
Set up your S3, Chef server, and IAM role as described here:
https://securosis.com/blog/using-amazon-iam-roles-to-distribute-security-credentials-for-chef
Execute “knife client ./” to create client.rb and validation.pem, then transfer them from your Chef server into your bucket.
Launch a new instance with the appropriate IAM Role you set up for Chef and your S3 bucket.
Specify your customized cloud-init script in the User Data field or command-line argument as described here:
https://securosis.com/blog/using-cloud-init-and-s3cmd-to-automatically-download-chef-credentials
You can also host the script as a file and load it from a central repository using an include.
Execute chef-client.