So I'm hosting an web app on EC2 which let users upload/download photos from S3. EC2 and S3 are in the same region.
Credentials: Following the official docs, credentials should be automatically supplied by the IAM role(with s3fullAccess permission) I created and associated to EC2.
Problem: User (from browser) can't download photos, but can successfully upload them. Download link was generated by aws-sdk's getSignedUrl() method.
Error Message:
InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
Firstly, access key seems correct as users can successfully upload stuffs.
My guess is that, since the user clicks link in the browser to download, there's no interaction between EC2 and S3, thus IAM role is not used when downloading. But isn't the signedDownloadUrl generated with EC2 credentials? And there's access key pairs supplied altogether in the query string. So my guess might be wrong..
Any suggestion/idea is appreciated.
Edit
I didn't create an IAM user, only have an IAM role. Don't know if this matters.
ASIA... is a temporary access key id from a set of temporary credentials for an IAM role. (Role credentials are always temporary.) AKIA... is an IAM user.
The problem here is that the x-amz-security-token accompanying the credentials was not used in the signing process that generated the S3 signed URL, so it doesn't appear in the signed URL... but it needs to be there for this to work.
If you are signing your request using temporary security credentials [...] you must include the corresponding security token in your request by adding the x-amz-security-token header.
http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#UsingTemporarySecurityCredentials
If you don't include x-amz-security-token, the system assumes it should look up the access-key-id in the IAM ussr database, and it won't be found there, which causes the InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records error.
Note that you can't simply add it to the already-generated URL. It needs to be incorporated into the signing process.
Related
I have a Laravel application that is hosted on AWS. I am using an S3 bucket to store files. I know that I have successfully connected to this bucket because when I upload files, they appear as I would expect inside the bucket's directories.
However, when I try to use the URL attached to the uploaded file to display it, I receive a 403 Forbidden error.
I have an IAM user set up named laravel which has the permission AmazonS3FullAccess applied to it, and I am using that key/secret.
I have the Object URL like so:
https://<BUCKET NAME>.s3.eu-west-1.amazonaws.com/<DIR>/<FILENAME>.webm
But if I try to access that either in my app (fed into an audio player) or just via the link directly, I get a 403. None of the tutorials I've followed to get this working involve Bucket Policies, but when I've googled the problems I'm having, Bucket Policy seems to come up.
Is there a single source of truth on how I am to do this? My AWS knowledge is very limited, but I am trying to get better!
When you request a URL of the form https://bucket.s3.amazonaws.com/dog/snoopy.png, that request is unauthenticated. Your S3 bucket policy does not allow unauthenticated access to the contents of the bucket so that request is denied with 403.
If you want your files to be downloadable by an unauthenticated/anonymous client then create an S3 bucket policy to allow that.
Alternatively, your server can create signed URLs and share those with the client.
Otherwise, your client's requests need to be authenticated, which means having correctly-permissioned credentials and using an AWS SDK.
Typically, back-end applications that you write that need access to data in S3 (or other AWS resources) would be given AWS credentials allowing the necessary access. If your back-end application runs in AWS then you would do that by launching the compute with an IAM role.
Typically, front-end applications would not have AWS credentials. Instead they would either authenticate to a back-end that then does work with AWS resources on their behalf. There are other options, however, such as AWS Amplify apps.
We're using a AWS Access Key/Secret pair for S3 uploads on an old client website. During an audit, we discovered that the Access Key used for uploads, while still working, doesn't appear to exist in any IAM user for the client's AWS account. I ran aws sts get-access-key-info --access-key-id=[old key] and it provided the correct AWS account id for our client. But searching for this key in our IAM users (https://console.aws.amazon.com/iam/home?#/users) shows no results. How can this be? Can the Access Key live somewhere else outside IAM?
Accepted answer from #kdgregory
It could belong to the root user. If yes, then you very much want to disable it. – kdgregory 2 hours ago
AFAIK AWS Access Key/Secret you can only secure and store once you created the IAM user. I don't think you can get the Access Key and Secret pair again anywhere in AWS unless you have it stored somewhere (which is not a good practice BTW).
I have provided AmazonS3FullAccess policy for both the IAM user and group. Also the buket that I am trying to access says "Objects can be public". I have explicitly made the folder inside the bucket public. Despite all this I am getting access denied error when I tried to access it through its url. Any idea on this?
Objects in Amazon S3 are private by default. This means that objects are not accessible by anonymous users.
You have granted permission for your IAM User to be able to access S3. Therefore, you have access to the objects but you must identify yourself to S3 so that it can verify your identity.
You should be able to access S3 content:
Via the Amazon S3 management console
Using the AWS CLI (eg aws s3 ls s3://bucketname)
Via authenticated requests in a web browser
I suspect that you have been accessing your bucket via an unauthenticated request (eg bucketname.s3.amazonaws.com/foo.txt. Unfortunately, this does not tell Amazon S3 who you are, so it will deny the request.
To access content with this type of URL, you can generate an Amazon S3 pre-signed URLs, which appends some authentication information to the URL to prove your identity. An easy way to generate the URL is with the AWS CLI:
aws s3 presign s3://bucketname/foo.txt
It will return a URL that looks like this:
https://bucketname.s3.amazonaws.com/foo.txt?AWSAccessKeyId=AKIAxxx&Signature=xxx&Expires=1608175109
The URL will be valid for one hour by default, up to 7 days.
There are two ways I will recommend.
go to s3 dashboard, and download the object you need, one by one manually, the bucket can be kept private at the same time.
build a gateway/a small service, to handle authentication for you, set a policy and give the permission to the service container/lambda to visit the private bucket, and restrict only specific users to download the objects.
References
download from aws s3
aws policy, permission and roles
I'm not sure if this is the appropriate use case, so please tell me what to look for if I'm incorrect in my assumption of how to do this.
What I'm trying to do:
I have an s3 bucket with different 'packs' that users can download. Upon their purchase, they are given a user role in Wordpress. I have an S3 browser set up via php that makes requests to the bucket for info.
Based on their 'role', it will only show files that match prefix (whole pack users see all, single product people only see single product prefix).
In that way, the server will be sending the files on behalf of the user, and changing IAM roles based on the user's permission level. Do I have to have it set that way? Can I just analyze the WP role and specify and endpoint or query that notes the prefixes allowed?
Pack users see /
Individual users see /--prefix/
If that makes sense
Thanks in advance! I've never used AWS, so this is all new to me. :)
This sounds too complex. It's possible to do with AWS STS but it would be extremely fragile.
I presume you're hiding the actual S3 bucket from end users and are streaming through your php application? If so, it makes more sense to do any role-based filtering in the php application as you have far more logic available to you there - IAM is granular, but restrictions to resources in S3 is going to be funky and there's always a chance you'll get something wrong and expose the incorrect downloads.
Rather do this inside your app:
establish the role you've granted
issue the S3 ls command filtered by the role - i.e. if the role permits only --prefix, issue the ls command so that it only lists files matching --prefix
don't expose files in the bucket globally - only your app should have access to the S3 bucket - that way people also can't share links once they've downloaded a pack.
this has the added benefit of not encoding your S3 bucket structure in IAM, and keeps your decision logic isolated to code.
There are basically three ways you can grant access to private content in Amazon S3.
Option 1: IAM credentials
You can add a policy to an IAM User, so that they can access private content. However, such credentials should only be used by staff in your own organization. it should not be used to grant access to application users.
Option 2: Temporary credentials via STS
Your application can generate temporary credentials via the AWS Security Token Service. These credentials can be given specific permissions and are valid for a limited time period. This is ideal for granting mobile apps access to Amazon S3 because they can communicate directly with S3 without having to go via the back-end app. The credentials would only be granted access to resources they are permitted to use.
These types of credentials can also be used by web applications, where the web apps make calls directly to AWS services (eg from Node/JavaScript in the browser). However, this doesn't seem suitable for your WordPress situation.
Option 3: Pre-Signed URLs
Imagine a photo-sharing application where users can access their private photos, and users can also share photos with other users. When a user requests access to a particular photo (or when the back-end app is creating an HTML page that uses a photo), the app can generate a pre-signed URL that grants temporary access to an Amazon S3 object.
Each pre-signed URL gives access only to a single S3 object and only for a selected time period (eg 5 minutes). This means that all the permission logic for whether a user is entitled to access a file can be performed in the back-end application. When the back-end application provides a pre-signed URL to the user's browser, the user can access the content directly from Amazon S3 without going via the back-end.
See: Amazon S3 pre-signed URLs
Your situation sounds suitable for Option #3. Once you have determined that a user is permitted to access a particular file in S3, it can generate the pre-signed URL and include it as a link (or even in <img src=...> tags). The user can then download the file. There is no need to use IAM Roles in this process.
I am using developer authenticated AWS cognito.
I want users to be able to create files under their own folders in S3.
I can do this using this policy variable:
${cognito-identity.amazonaws.com:sub}
EXCEPT it's not my developer issued userid! It is Amazon's userid which is a long string of characters. Not what I want as foldernames in S3.
I can't find how to get the correct policy variable to use in the policy that gives access to S3.
Unfortunately the developer identifier is not exposed to an IAM policy, only the identity id can currently be used for that. The LookupDeveloperIdentity API could be used to get the identity id associated with your developer identifier to name the S3 folder, though.