I encounter a SignNotMatch when I generate a presigned url with boto3 with the code below:
session = Session(access_key, secret_key)
s3 = session.client('s3', endpoint_url=OSS_ENDPOINT, config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': bucket,
'Key': key
}
)
And then parse the request and resign(use AWS Go SDK signer.Presign) it in our proxy, and I always get a not match error.
Then I open the debug mode in boto3 and add log in AWS Go SDK, and found that when they calculate Canonical Request they use different way:
Canonical Request:
HTTP Verb + '\n' +
Canonical URI + '\n' +
Canonical Query String + '\n' +
Signed Headers + '\n' +
"UNSIGNED-PAYLOAD"
In Go AWS SDK it will put X-Amz-Content-Sha256=UNSIGNED-PAYLOADin Canonical Query String by default while boto3 will not.
Is it supposed to or I use it in a wrong way?
Related
I am struggling to read .m3u8 file in Javascript and modify its segments to get signed by Cloudfront for streaming hls content.
const s3ObjectKey = `${folderName}/${folderName}.m3u8`;
const url = `${process.env.CLOUDFRONT_DOMAIN}/${s3ObjectKey}`;
const privateKey = fs.readFileSync(
new URL("../private_key.pem", import.meta.url),
{
encoding: "utf8",
}
);
const keyPairId = process.env.CLOUDFRONT_KEY_PAIR_ID;
const dateLessThan = new Date(new Date().getTime() + 60 * 60000);
const m3u8Url = cloudfrontSignedUrl({
url,
keyPairId,
dateLessThan,
privateKey,
});
After I get the signed m3u8Url I need to modify its segments to be signed.
Any help will be appreciated.
Answer form AWS Blog post
import boto3
import os
KEY_PREFIX = os.environ.get('KEY_PREFIX')
S3_BUCKET = os.environ.get('S3_BUCKET')
SEGMENT_FILE_EXT = os.environ.get('SEGMENT_FILE_EXT', '.ts')
required_vars = [KEY_PREFIX, S3_BUCKET]
if not all(required_vars):
raise KeyError(f'Missing required environment variable/s. Required vars {required_vars}.')
s3 = boto3.client('s3')
def lambda_handler(event, context):
try:
s3_key = event['pathParameters']['proxy']
obj = s3.get_object(Bucket=S3_BUCKET, Key=s3_key)
body = obj['Body'].read().decode('utf-8')
qp = event['queryStringParameters']
params = ['?']
# reconstruct query param uri
[(params.append(p.replace(KEY_PREFIX, '') + '=' + qp[p] + "&")) for p in qp if KEY_PREFIX in p]
sign_params = ''.join(params).rstrip("&")
# append query params to each segment
resp_body = body.replace(SEGMENT_FILE_EXT, ''.join([SEGMENT_FILE_EXT, sign_params]))
return {
'statusCode': 200,
'body': resp_body
}
except Exception as e:
print(e)
return {'statusCode': 500, 'body': ''}
Let’s go over the key areas in the python code:
KEY_PREFIX environment variable is the prefix pattern that Lambda uses to identify the CloudFront query params. In my example it is -PREFIX.
S3_BUCKET environment variable the S3 bucket name to which Lambda will make the request to get the main manifest.
Note: For illustration purposes I used an environment variable that
is set, for instance, when the Lambda function is created. You can
change this part to have a lookup logic, which is especially helpful
if you have different S3 buckets that are used to store media
content.
SEGMENT_FILE_EXT environment variable file extension of your media file. It default’s to .ts, you can override the value by setting this environment variable in Lambda function configuration.
s3_key variable is the path to the file in your S3 bucket which is represented as a URL that is used by the client to make the request. In my example the URL is https://myapp.com/movies/movie_1/index.m3u8 and the s3_key is movies/movie_1/index.m3u8, which matches exactly with the s3 bucket folder structure as shown earlier in Figure 5. By following this convention, when creating a S3 folder structure for the URL path, s3_key will dynamically get the correct path to the file in your S3 bucket. To learn more about Lambda proxy integration, see Set up a proxy integration with a proxy resource.
sign_params variable is the reconstructed CloudFront signed URL query parameters.
resp_body variable, is the final modified main manifest that is returned to the client. The replace function appends the CloudFront signed URL query parameters to each segment in the manifest file. The final result is assigned to the resp_body variable.
This Lambda function is getting the manifest file in its original form from your S3 bucket and modifying it so that when the playback player makes a request to get the next segment, the request already includes the CloudFront signed URL query parameters. This allows you to restrict access to the video content based on user permissions for each video. Figure 6 illustrates manifest before and after the modification.
I'm trying to provide a pre-signed url that, once the image is uploaded, grants the group Everyone read access to the uplodaded image.
So far, I'm generating the pre-signed url with the following steps:
val req = GeneratePresignedUrlRequest(params.s3Bucket,"$uuid.jpg",HttpMethod.PUT)
req.expiration = expiration
req.addRequestParameter("x-amz-acl","public-read")
req.addRequestParameter("ContentType","image/jpeg")
val url: URL = s3Client.generatePresignedUrl(req)
But the image, once I check in S3, does not have the expected read access.
The HTTP client that performs the upload needs to include the x-amz-acl: public-read header.
In your example, you're generating a request that includes that header. But, then you're generating a presigned URL from that request.
URLs don't contain HTTP headers, so whatever HTTP client you're using to perform the actual upload is not sending setting the header when it sends the request to the generated URL.
This simple answer is working for me.
val url = getS3Connection()!!.generatePresignedUrl(
"bucketname", "key",
Date(Date().time + 1000 * 60 * 300)
)
There are a few examples for the way to pre-sign the URL of an S3 request, but I couldn't find any working example to pre-sign other services in AWS.
I'm trying to write an item to DynamoDB using the Python SDK botos. The SDK included the option to generate the pre-signed URL here. I'm trying to make it work and I'm getting a URL, but the URL is responding with 404 and the Item is not appearing in the DynamoDB table.
import json
ddb_client = boto3.client('dynamodb')
response = ddb_client.put_item(
TableName='mutes',
Item={
'email': {'S':'g#g.c'},
'until': {'N': '123'}
}
)
print("PutItem succeeded:")
print(json.dumps(response, indent=4))
This code is working directly. But when I try to presign it:
ddb_client = boto3.client('dynamodb')
params = {
'TableName':'mutes',
'Item':
{
'email': {'S':'g#g.c'},
'until' : {'N': '1234'}
}
}
response = ddb_client.generate_presigned_url('put_item', Params = params)
and check the URL:
import requests
r = requests.post(response)
r
I'm getting: Response [404]
Any hint on how to get it working? I checked the IAM permissions, and they are giving full access to DynamoDB.
Please note that you can sign a request to DynamoDB using python, as you can see here: https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html#sig-v4-examples-post . But for some reasons, the implementation in the boto3 library doesn't do that. Using the boto3 library is much easier than the code above, as I don't need to provide the credentials for the function.
You send an empty post request. You should add the data to the request:
import requests
r = requests.post(response, data = params)
I think you are having this issue, that's why you are recieving a 404.
They recommend using Cognito for authentication instead of IAM for this cases.
I have seen pre-signed URL for S3 object. Is it possible to create pre-signed URL for API gateway. I have gone through documentation. I am using .NET. I would like to know if there is .NET library available to create pre-signed request for gateway API.
ISSUE
I have GET API something like this https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/pets?type=dog&page=1 and our client is going to invoke that API once in a while. The legacy tool that they are using only supports GET. So i wanted to create a pre-signed URL (with short expiry time) and give them when they ask for it. For each client i already have IAM user with their respective accesskey and secretkey
PreSigned URLs are typically signed with AWS SigV4 signing process.
You can generate SigV4 signed Urls for your API Gateway Hosted Endpoints. Typically, you will need to send SigV4 signature in Authorization Request Header. If you are clients are willing to send header, here is one sample Library you can try for .NET which creates a HTTP Request with signed header.
If your clients cannot send Authorization Header or cannot use above library then you can convert the signature to be a Query String Format and provide the pre-signed Urls to them.
This AWS Documentation has example in Python on how to generate Query String URL. Now, you can take python example and convert into .NET based code with following sample.
public string GetSig4QueryString(string host, string service, string region)
{
var t = DateTimeOffset.UtcNow;
var amzdate = t.ToString("yyyyMMddTHHmmssZ");
var datestamp = t.ToString("yyyyMMdd");
var canonical_uri = "/dev/myApigNodeJS";
var canonical_headers = "host:" + host+"\n";
var signed_headers = "host";
var credential_scope = $"{datestamp}/{region}/{service}/aws4_request";
var canonical_querystring = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=" + WebUtility.UrlEncode(_access_key + "/" + credential_scope)
+ "&X-Amz-Date=" + amzdate + "&X-Amz-SignedHeaders=" + signed_headers;
Console.WriteLine("canonical_querystring");
Console.WriteLine(canonical_querystring);
var payload_hash = Hash(new byte[0]);//No Payload for GET
var canonical_request = new StringBuilder();
canonical_request.Append("GET\n");
canonical_request.Append(canonical_uri + "\n");
canonical_request.Append(canonical_querystring + "\n");
canonical_request.Append(canonical_headers + "\n");
canonical_request.Append(signed_headers + "\n");
canonical_request.Append(payload_hash);
Console.WriteLine("canonical_request");
Console.WriteLine(canonical_request);
var string_to_sign = $"{algorithm}\n{amzdate}\n{credential_scope}\n" + Hash(Encoding.UTF8.GetBytes(canonical_request.ToString()));
Console.WriteLine("string_to_sign");
Console.WriteLine(string_to_sign);
var signing_key = GetSignatureKey(_secret_key, datestamp, region, service);
var signature = ToHexString(HmacSHA256(signing_key, string_to_sign));
var signed_querystring = canonical_querystring+"&X-Amz-Signature=" + signature;
return signed_querystring;
}
GetSig4QueryString("myApiId.execute-api.us-east-1.amazonaws.com","execute-api","us-east-1");
//Returned String --> X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Full Endpoint Becomes -
https://myApiId.execute-api.us-east-1.amazonaws.com/dev/myApigNodeJS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Note -
This example code refers methods & variables from Github project I gave above.
Also, this example hard coded API Path /dev/myApigNodeJS and signs it and it will be different for you with full absolute path.
AWS recommends to sign all queryStrings, headers which you are planning to send in request. Go through .NET code of library I referred and understand how its doing that.
Let me know if you have questions.
When generating the presigned url a websocket service within the AWS API Gateway, I used the solution by Imran and added the "X-Amz-Security-Token" which is required.
I create a pre-signed URL and get back something like
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
I can now curl this no problem. However, if I now add another query parameter, I will get back a 403, i.e.
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
&Foo=123
How come? Is it possible to generate a pre-signed url that supports custom queries?
It seems to be technically feasible to insert custom query parameters into a v4 pre-signed URL, before it is signed, but not all of the AWS SDKs expose a way to do this.
Here's an example of a roundabout way to do this with the AWS JavaScript SDK:
const AWS = require('aws-sdk');
var s3 = new AWS.S3({region: 'us-east-1', signatureVersion: 'v4'});
var req = s3.getObject({Bucket: 'mybucket', Key: 'mykey'});
req.on('build', () => { req.httpRequest.path += '?session=ABC123'; });
console.log(req.presign());
I've tried this with custom query parameters that begin with X- and without it. Both appeared to work fine. I've tried with multiple query parameters (?a=1&b=2) and that worked too.
The customized pre-signed URLs work correctly (I can use them to get S3 objects) and the query parameters make it into CloudWatch Logs so can be used for correlation purposes.
Note that if you want to supply a custom expiration time, then do it as follows:
const Expires = 120;
const url = req.presign(Expires);
I'm not aware of other (non-JavaScript) SDKs that allow you to insert query parameters into the URL construction process like this so it may be a challenge to do this in other languages. I'd recommend using a small JavaScript Lambda function (or API Gateway plus Lambda function) that would simply create and return the customized pre-signed URL.
The custom query parameters are also tamper-proof. They are included in the signing of the URL so, if you tamper with them, the URL becomes invalid, yielding 403 Forbidden.
I used this code to generate your pre-signed URL. The result was:
https://s3.amazonaws.com/MyBucket/MyItem
?Foo=123
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA...27%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20180427T0012345Z
&X-Amz-Expires=3600
&X-Amz-Signature=e3...7b
&X-Amz-SignedHeaders=host
None of this is a guarantee that this technique will continue to work, of course, if AWS changes things under the covers but for right now it seems to work and is certainly useful.
Attribution: the source of this discovery was aws-sdk-js/issues/502.
If you change one of the headers or add / subtract, then you have to resign the URL.
This is part of the AWS signing design and this process is designed for higher levels of security. One of the AWS reasons for changing to signing version 4 from signing version 2.
The signing design does not know which headers are important and which are not. That would create a nightmare trying to track all of the AWS services.
I created this solution for Ruby SDK. It is sort of a hack, but it works as expected:
require 'aws-sdk-s3'
require 'active_support/core_ext/object/to_query.rb'
# Modified S3 pre signer class that can inject query params to the URL
#
# Usage example:
#
# bucket_name = "bucket_name"
# key = "path/to/file.json"
# filename = "download_file_name.json"
# duration = 3600
#
# params = {
# bucket: bucket_name,
# key: key,
# response_content_disposition: "attachment; filename=#{filename}",
# expires_in: duration
# }
#
# signer = S3PreSignerWithQueryParams.new({'x-your-custom-field': "banana", 'x-some-other-field': 1234})
# url = signer.presigned_url(:get_object, params)
#
# puts "url = #{url}"
#
class S3PreSignerWithQueryParams < Aws::S3::Presigner
def initialize(query_params = {}, options = {})
#query_params = query_params
super(options)
end
def build_signer(cfg)
signer = super(cfg)
my_params = #query_params.to_h.to_query()
signer.define_singleton_method(:presign_url,
lambda do |options|
options[:url].query += "&" + my_params
super(options)
end)
signer
end
end
While not documented, you can add parameters as arguments to the call to presigned_url.
obj.presigned_url(:get,
expires_in: expires_in_sec,
response_content_disposition: "attachment"
)
https://bucket.s3.us-east-2.amazonaws.com/file.txt?response-content-disposition=attachment&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=PUBLICKEY%2F20220309%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220309T031958Z&X-Amz-Expires=43200&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATUREVALUE
If you are looking on for JavaScript SDK V3:
import { HttpRequest } from "#aws-sdk/protocol-http";
import { S3RequestPresigner } from "#aws-sdk/s3-request-presigner";
import { parseUrl } from "#aws-sdk/url-parser";
import { Sha256 } from "#aws-crypto/sha256-browser";
import { Hash } from "#aws-sdk/hash-node";
import { formatUrl } from "#aws-sdk/util-format-url";
// Make custom query in Record<string, string | Array<string> | null> format
const customQuery = {
hello: "world",
};
const s3ObjectUrl = parseUrl(
`https://${bucketName}.s3.${region}.amazonaws.com/${key}`
);
s3ObjectUrl.query = customQuery; //Insert custom query here
const presigner = new S3RequestPresigner({
credentials,
region,
sha256: Hash.bind(null, "sha256"), // In Node.js
//sha256: Sha256 // In browsers
});
// Create a GET request from S3 url.
const url = await presigner.presign(new HttpRequest(s3ObjectUrl));
console.log("PRESIGNED URL: ", formatUrl(url));
Code template taken from: https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/