I'm kinda new to AWS S3 - using EC2 (hosting web app) and S3 (storing resources) in the same AWS region, and assigned EC2 with an IAM role s3access, so EC2 can download from S3 easily.
The question is, how should a client download from S3? Apparently the client doesn't have IAM role or Access Key like EC2 does. Seems the client only have a signedDownloadUrl generated by aws-sdk, but it also requires access key.
So, should I make the bucket public then any client can download, or should I find some approach to supply the client with credentials?
All objects by default are private. Only the object owner has permission to access these objects.
So if you want to share an object with someone you can
You can make it public or
You, the object owner can share objects with others by creating a pre-signed URL, using their own security credentials, to grant time-limited permission to download the objects.
For more details on pre-signed URLs refer S3 Share Objects with PreSignedURL
Related
Use Case:
I want to be able to:
Upload images and audio files from my backend to S3 bucket
List and view/play content on my backend
Return the objects URLs in API responses
Mobile apps can view/play the URLs with or without? authentication from the mobile side
Is that possible without making the S3 bucket public ?
Is that possible without making the S3 bucket public ?
Yes, it should be possible. Since you are using EC2 instance for backend, you, you can setup instance role to enable private and secure access of your backed application to S3. In the role, you would allow S3 read/write. This way, if your application is using AWS SDK, you can seamlessly access S3 without making S3 public.
Regarding the links to the object, the general way is to return S3 pre-signed links. This allows for temporary access to your objects without the need for public access. The alternative is to share your objects through CloudFront as explained in Amazon S3 + Amazon CloudFront: A Match Made in the Cloud. In either case, bucket can be private.
I have provided AmazonS3FullAccess policy for both the IAM user and group. Also the buket that I am trying to access says "Objects can be public". I have explicitly made the folder inside the bucket public. Despite all this I am getting access denied error when I tried to access it through its url. Any idea on this?
Objects in Amazon S3 are private by default. This means that objects are not accessible by anonymous users.
You have granted permission for your IAM User to be able to access S3. Therefore, you have access to the objects but you must identify yourself to S3 so that it can verify your identity.
You should be able to access S3 content:
Via the Amazon S3 management console
Using the AWS CLI (eg aws s3 ls s3://bucketname)
Via authenticated requests in a web browser
I suspect that you have been accessing your bucket via an unauthenticated request (eg bucketname.s3.amazonaws.com/foo.txt. Unfortunately, this does not tell Amazon S3 who you are, so it will deny the request.
To access content with this type of URL, you can generate an Amazon S3 pre-signed URLs, which appends some authentication information to the URL to prove your identity. An easy way to generate the URL is with the AWS CLI:
aws s3 presign s3://bucketname/foo.txt
It will return a URL that looks like this:
https://bucketname.s3.amazonaws.com/foo.txt?AWSAccessKeyId=AKIAxxx&Signature=xxx&Expires=1608175109
The URL will be valid for one hour by default, up to 7 days.
There are two ways I will recommend.
go to s3 dashboard, and download the object you need, one by one manually, the bucket can be kept private at the same time.
build a gateway/a small service, to handle authentication for you, set a policy and give the permission to the service container/lambda to visit the private bucket, and restrict only specific users to download the objects.
References
download from aws s3
aws policy, permission and roles
Scenario: A web app is hosted on EC2 with a role having full S3 access. Now on the webapp, say the S3 resource is an image embedded as a link in some webpage.
So here CORS comes in to action. So if CORS and Public access for the resource are enabled (for the Bucket I mean), that should do.
And when public access is already granted, what's the point in adding a role to EC2 instance, the resource is already public.
So, is not the role for EC2 instance redundant and not necessary. Could someone confirm or suggest the right permissions to use?
(Am required to ask for this clarification as few suggestions on StackOverflow mentioned that role for EC2 is required.)
First, I should mention that use of the AmazonS3FullAccess policy should be avoided if possible. It grants permission to do anything in Amazon S3, including deleting all buckets and objects! Always grant least privilege when possible, only granting the access that is necessary.
If the app is linking to public objects, then no additional S3 permissions are required for the scenario you describe.
However, many apps do not with to make all files public. For example, let's say the app is a photo storage/sharing app. People would like to keep their photos private, yet be able to view them in a web page. To do this, the app can generate an Amazon S3 pre-signed URL that provides temporary access to private objects in Amazon S3. This way, an object can be referenced in an <img> tag on an HTML page, yet remain private.
Creating a pre-signed URL requires a set of credentials that have, at minimum, read access to the Amazon S3 object being shared. Thus, it would be necessary for the app to receive these credentials (normally via an IAM Role).
If the files are public, there is no need to also give s3fullaccess permissions to the role - public is public; giving s3fullaccess would make your app less secure in fact, because you are giving it permissions beyond what it needs to run, which is never a good idea.
In one of the blog post, the author has mentioned that he uploaded dataset into a s3 bucket and gave public access.
s3://us-east-1.elasticmapreduce.samples/flightdata/input
Now I want to download/see the data from my chrome browser.
When I copy paste above link in chrome address bar it is asking for:
Access key ID
Secret access key
What should I give here?
Did the author initially made it public and now made it private?
(I am confused)
Also can we access these kind of URLs that start with s3:// directly from browsers?
Should I need to have a AWS account to access these S3 buckets?
(I know we can access web data using http protocol.. http://)
The Amazon S3 management console allows you to view buckets belonging to your account. It is not possible to view S3 buckets belonging to other accounts within the S3 console.
You can, however, access them via the AWS Command-Line Interface (CLI). For example:
aws s3 ls s3://us-east-1.elasticmapreduce.samples/flightdata/input/
You can also copy files from other buckets by using aws s3 cp and aws s3 sync.
These calls require a set of valid AWS credentials (Access Key and Secret Key), which can be stored in the credentials files via the aws configure command. You do not need specific permission to access public buckets, but you do need permission to use S3 in general. You can obtain an Access Key and Secret Key in the IAM management console where your IAM User is defined. (Or, if you do not have permission to view it, ask your AWS administrator for the Access Key and Secret Key.)
Can you connect to S3 via s3cmd or mount S3 to and ec2 instance with IAM users and not using access keys?
All the tutorials I see say to use access keys but what if you can't create your own access keys (IT policy).
There are two ways to access data in Amazon S3: Via an API, or via URLs.
Via an API
When accessing Amazon S3 via API (which includes code using an AWS SDK and also the AWS Command-Line Interface (CLI)), user credentials must be provided in the form of an Access Key and a Secret Key.
The aws and s3cmd utilities, and also software that mounts Amazon S3 as a drive, require access to the API and therefore require credentials.
If you have been given a login to an AWS account, you should be able to ask your administrators to also create credentials that are associated with your User. These credentials will have exactly the same permissions as your normal login (via Username/password), so it's strange that they would be disallowing it. They can be very useful for automating AWS activities, such as starting/stopping Amazon EC2 instances.
Via URLs
Objects stored in Amazon S3 can also be made available via a URL that points directly to the data, eg s3.amazonaws.com/bucket-name/object.txt
To provide public access to these objects without requiring credentials, either add permission to each object or create a Bucket Policy that grants access to content within the bucket.
This access method can be used to retrieve individual objects, but is not sufficient to mount Amazon S3 as a drive.