I'm trying to generate a presigned upload URL to AWS S3, and this is the URL that I receive:
iex(3)> ExAws.S3.presigned_url(ExAws.Config.new(:s3), :put, "myapp-staging", "picture.png")
{:ok, "https://s3.us-east-1.amazonaws.com/myapp-staging/picture.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MY-KEY-IS-HERE%2F20191215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191215T220120Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=df6a289130d12e3e8175457b5f8e21a8840f9b5713134ac83c91368332b84eed"}
Notice MY-KEY-IS-HERE - the AWS access key id is part of the url generated.
Is it safe for this API key to be publicly visible to anyone online asking for a presigned upload URL?
ACCESS_KEY_ID is ok but SECRET ACCESS Key will be cause the security issue.
Related
Have following code to generate pre-signed URL:
params = {'Bucket': bucket_name, 'Key': object_name}
response = s3_client.generate_presigned_url('get_object',
Params=params,
ExpiresIn=expiration)
that works fine on old one bucket I am using for last year:
https://old-bucket.s3.amazonaws.com/test_image.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIxxxxxxxxxxE%2F20210917%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20210917T210448Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=54e173601fec5f140dd901b0eae1dafbcd8d7ee8b8f311fdc1b120ca447cdd0c
I can paste this URL to browser and download file. File is AWS-KMS encrypted.
But same AWS-KMS encrypted file uploaded to new one created bucket returns following URL:
https://new-bucket.s3.amazonaws.com/test_image.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIxxxxxxxxxxE%2F20210917%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20210917T210500Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2313e0131d4251f9fba522fc8e9880d960f674f3449e141848bd38ca19e1b528
returns SignatureDoesNotMatch error:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
No any changes in source code - but just bucket name provided to generate_presigned_url function.
The IAM user I am providing to boto3.client has write/read permissions for both buckets.
Comparing properties and permissions for both buckets and for files I am requesting from buckets - everything looks the same.
GetObject and PutObject works fine for both buckets in a case of dealing with file directly. The issue is only in a case of using pre-signed URL.
So is any settings/permissions/rules/anything else need to be configured/enabled to make pre-signed URLs working with certain S3 bucket?
I created an IAM role that gives full access to the S3 Bucket and attached it to the EC2 instance. However, I am unable to view the image when I try to view it from the EC2 hosted website. I keep getting a 403 Forbidden code.
Below is the IAM role and the policy attached:
It is seen that GetObject is enabled:
But the error still persists:
Any advice on how to solve this? Thank you for reading.
The URL you are using to access the object does not appear to include any security information (bucket.s3.amazonaws.com/cat1.jpg). Thus, it is simply an 'anonymous' request to S3, and since the object is private, S3 will deny the request.
The mere fact that the request is being sent from an Amazon EC2 instance that has been assigned an IAM Role is not sufficient to obtain access to the object via an anonymous URL.
To allow a browser to access a private Amazon S3 object, your application should generate an Amazon S3 pre-signed URLs. This is a time-limited URL that includes security information that identifies you as the requester and includes a signature that permits access to the private object.
Alternatively, code running on the instance can use an AWS SDK to make an API call to S3 to access the object (eg GetObject()). This will succeed because the AWS SDK will use the credentials provided by the IAM Role.
Following is what I'm doing. I'm generating a pre-signed URL using a custom domain for my s3 bucket resources which are not public.
https://files.customdomain.com/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
Also to add the certificate I've created a cloudfront distribution for the bucket having following origin settings
Origin Domain Name: bucket-name.s3.amazonaws.com
Origin Id : s3.bucket-name
Restrict Bucket Access: No
Yet I'm unable to access my resources. Throws access denied error. Any help would be appreciated.
There are two cases:
If your bucket has regular name.
In this case you should use CloudFront to access your bucket.
And like mentioned above URL looks like in this answer:
https://cloudfront-url/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
If your bucket has s3 static website name.
In this case your bucket name looks like files.customdomain.com and you can generate pre-signed url for this bucket:
https://files.customdomain.com/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
In your DNS you will have CNAME files.customdomain.com points to files.customdomain.com.s3.[bucket-region].amazonaws.com.
NOTICE
When I generate pre-signed URL via aws-cli:
aws s3 presign s3://files.customdomain.com/file123 --endpoint-url https://files.customdomain.com
I get URL with duplicate bucket name in the path:
https://files.customdomain.com/files.customdomain.com/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
instead of:
https://files.customdomain.com/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
I don't know if it has the same behavior via SDK.
Have you tried initializing S3 with the custom url var S3 = new AWS.S3({endpoint: 'media.domain.com', s3BucketEndpoint: true});
More info https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
Also, make sure signature is correct as well https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version
Ref : https://github.com/aws/aws-sdk-js/issues/891
When using S3 with CloudFront, you don't want an S3 signed URL... you want a CloudFront signed URL.
Read Configuring Security and Limiting Access to Content in the CloudFront developer guide.
I found a solution for this question. The signed url needs to be generated for cloudfront url endpoint from s3 bucket. Therefore instead of
https://files.customdomain.com/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
it needs to be
https://cloudfront-url/file123?AWSAccessKeyId=XXX&Expires=1541220685&Signature=XXXX
and DNS records had to resolve custom domain to cloudfront url.
I am wondering if it is possible to use pre-signed urls with other aws services other than s3. Specifically, the boto3 documentation http://boto3.readthedocs.io/en/latest/reference/services/logs.html#CloudWatchLogs.Client.generate_presigned_url shows that the method generate_presigned_url is available for cloudwatch logs. I've tried using it in the following fashion.
client = boto3.client(
'logs',
aws_access_key_id="<aws_access_key_id>",
aws_secret_access_key="<aws_secret_access_key>",
region_name='us-east-1'
)
url = client.generate_presigned_url(
ClientMethod='get_log_events',
Params={
'logGroupName':'<logGroupName>',
'logStreamName':'<logStreamName>'
},
ExpiresIn=180
)
The url generates, but when trying to access the url I get the error in the browser:
<InvalidSignatureException>
<Message>The request signature we calculated does not match the
signature you provided. Check your AWS Secret Access Key and signing
method. Consult the service documentation for details.</Message>
</InvalidSignatureException>
For reference, the url is in this format (AWS Signature Version 4):
https://logs.us-east-1.amazonaws.com/
?logGroupName=<logGroupName>&logStreamName=<logStreamName>
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=<aws_access_key_id>%2F20130721%2Fus-east-
1%2Fs3%2Faws4_request
&X-Amz-Date=20180531T150510Z
&X-Amz-Expires=180
&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-target
&X-Amz-Signature=<signature-value>
How can I access this url? I noticed this url has a different format than the ones generated for s3, which works with the same method (i.e. generate_presigned_url with get_objects). Is there a way to make this work with Cloudwatch Logs?
I have multiple images within a private S3 bucket and I would like an instance of Tableau to be able to access those images. Is there a URL or some way to access those images while still keeping the S3 bucket private?
Access Private Bucket through Tableau
You can setup a IAM user with access permission to S3 and allow Tableau access.
Check the article on Connect to your S3 data with the Amazon Athena connector in Tableau 10.3 for more details.
Note: You need to configure Amazon Athena for Querying the S3 content.
Custom Generated S3 Urls to Access Private Bucket
Yes. You can generate a Signed URL from your backend using AWS SDK. This can be done directly using S3 or through AWS CloudFront.
Using S3 Signed Urls. e.g, Signed Url for GET Object.
var params = {Bucket: 'bucket', Key: 'key'};
var url = s3.getSignedUrl('getObject', params);
console.log('The URL is', url);
Using CloudFront Signed Urls. e.g, Signed Url for GET in CloudFront.
var cfsign = require('aws-cloudfront-sign');
var signingParams = {
keypairId: process.env.PUBLIC_KEY,
privateKeyString: process.env.PRIVATE_KEY,
// Optional - this can be used as an alternative to privateKeyString
privateKeyPath: '/path/to/private/key',
expireTime: 1426625464599
}
// Generating a signed URL
var signedUrl = cfsign.getSignedUrl(
'http://example.cloudfront.net/path/to/s3/object',
signingParams
);
Note: Generating the Url needs to be done in your backend. You can setup a serverless solution for this by using AWS API Gateway and Lambda to provide an endpoint for authenticated users to access.
In addition you can also use AWS Cognito UserPools with Identity Pool to get direct access to S3 Private Content without the above steps. For this you need to use the Cognito UserPools or a federated identity as the identity provider which is connected with Cognito Identity Pools.