I am trying to run a script from OpenTraffic repository, and it needs access to some AWS S3 buckets. I am unable to figure out how to get access to a particular AWS S3 bucket?
FYI:
OpenTraffic is a open source platform to obtain and analyse dynamic traffic data : https://github.com/opentraffic
The script I am trying to run:
https://github.com/opentraffic/reporter/blob/dev/load-historical-data/load_data.sh
Documentation(https://github.com/opentraffic/reporter/tree/dev/load-historical-data) says: In order to run above script,
access required to both s3://grab_historical_data, s3://reporter-drop-
{prod, dev}.
Your're accessing the S3 buckets from r3.4xlarge ec2 instance according to the documentation link your share.
Firstly, You've to create a IAM role for ec2 instance and S3 access policy with it.
Create the ec2 instance and attach the IAM role to it because this is the only time you can to assign a role to it and launch it.
Role gives your ec2 instance access permission for s3 bucket.
Related
So I have created an IAM user and added a permission to access S3 then I have created an EC2 instance and SSH'ed into the it.
After giving "aws s3 ls" command, the reply was
"Unable to locate credentials. You can configure credentials by running "aws configure".
so what's the difference between giving IAM credentials(Key and Key ID) using "aws configure" and editing the bucket policy to allow s3 access to my instance's public IP.
Even after editing the bucket policy(JSON) to allow S3 access to my instance's public IP why am I not able to access the s3 bucket unless I use "aws configure"(Key and Key ID)?
Please help! Thanks.
Since you are using EC2 you should really use EC2 Instance Profiles instead of running aws configure and hard-coding credentials in the file system.
As for the your question of S3 bucket policies versus IAM roles, here is the official documentation on that. They are two separate tools you would use in securing your AWS account.
As for your specific command that failed, note that the AWS CLI tool will always try to look for credentials by default. If you want it to skip looking for credentials you can pass the --no-sign-request argument.
However, if you were just running aws s3 ls then that was trying to list all the buckets in your account, which you would have to have IAM credentials for. Individual bucket policies would not be taken into account in that scenario.
If you were running aws s3 ls s3://bucketname then that may have worked as aws s3 ls s3://bucketname --no-sign-request.
When you create iam user so there are two parts
policies
roles
Policies are attached to user, like what all services user can pr can't access
roles are attached to application, what all access that application can have
So you have to permit ec2 to access S3
There are two ways for that
aws configure
attach role to ec2 instance
while 1 is tricky and legthy , 2 is easy
Go to ec2-instance-> Actions -> Security -> Modify IAM role -> then select role (ec2+s3 access role)
thats it , you can simply do aws s3 ls from ec2 instance
I cannot make the S3 bucket public. I would like a method to use without AWS command line interface.
You need to create a role with S3 access (full or limited, you decide) and attach this role to the EC2 instance. Now you can manipulate files in the S3 from the EC2.
We have not given anyone any download rights through S3 but it is still possible to download data through an EMR cluster using scp
Is it possible to give someone the cluster dns but make sure they can use the data in the cluster but not download it?
EMR nodes by default will assume EC2 instance profile: EMR_EC2_DefaultRole IAM role to access resources on your account including S3. Policies defined in this role will decide on what EMR had access to.
If that role has s3:* , or s3:get* etc, allowed , on all resources like buckets and objects, then all nodes on EMR can download objects from all buckets on your account. (Given you do not have any bucket policies).
http://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html https://aws.amazon.com/blogs/security/iam-policies-and-bucket-policies-and-acls-oh-my-controlling-access-to-s3-resources/
Yes, given EMR has access to S3 , if you are sharing the Private SSH key (.pem) file of an EMR/Ec2 with a user, they can use SCP to copy data from EMR to their machine.
I am mounting an AWS S3 bucket as a filesystem using s3fs-fuse. It requires a file which contains AWS Access Key Id and AWS Secret Access Key.
How do I avoid the access using this file? And instead use AWS IAM roles?
As per Fuse Over Amazon document, you can specify the credentials using 4 methods. If you don't want to use a file, then you can set AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables.
Also, if your goal is to use AWS IAM instance profile, then you need to run your s3fs-fuse from an EC2 instance. In that case, you don't have to set these credential files/environment variables. This is because while creating the instance, if you attach the instance role and policy, the EC2 instance will get the credentials at boot time. Please see the section 'Using Instance Profiles' in page 190 of AWS IAM User Guide
there is an argument -o iam_role=--- which helps you to avoid AccessKey and SecretAccessKey
The Full steps to configure this is given below
https://www.nxtcloud.io/mount-s3-bucket-on-ec2-using-s3fs-and-iam-role/
I have a question on using IAM roles with EC2 and EMR. Here's my current setup:
I have a EC2 machine launched with a particular IAM role (let's call this role 'admin'). My workflow is to upload a file to S3 from this machine and then create an EMR cluster with a particular IAM role (a 'runner' role). The EMR cluster works on the file uploaded to S3 from the admin machine.
Admin is a role with privileges to all APIs in all AWS services. Runner has access to all APIs in EMR, EC2 and S3.
For some reason, the EMR cluster is unable to access the input file loaded in S3. It keeps getting an 'access denied' exception from s3.
I guess writing to s3 from one IAM role and reading it from a different IAM role is what is causing the issue.
Any ideas on what is going wrong here or whether this is even a supported use-case is appreciated.
Thanks!
http://blogs.aws.amazon.com/security/post/TxPOJBY6FE360K/IAM-policies-and-Bucket-Policies-and-ACLs-Oh-My-Controlling-Access-to-S3-Resourc
S3 objects are protected in three ways as seen in the post I linked to.
Your IAM role will need the permission to read S3 objects.
The S3 bucket policy must allow your IAM role access to the object.
The S3 ACL for the specific object must also allow your IAM role access to the object.