I have seen pre-signed URL for S3 object. Is it possible to create pre-signed URL for API gateway. I have gone through documentation. I am using .NET. I would like to know if there is .NET library available to create pre-signed request for gateway API.
ISSUE
I have GET API something like this https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/pets?type=dog&page=1 and our client is going to invoke that API once in a while. The legacy tool that they are using only supports GET. So i wanted to create a pre-signed URL (with short expiry time) and give them when they ask for it. For each client i already have IAM user with their respective accesskey and secretkey
PreSigned URLs are typically signed with AWS SigV4 signing process.
You can generate SigV4 signed Urls for your API Gateway Hosted Endpoints. Typically, you will need to send SigV4 signature in Authorization Request Header. If you are clients are willing to send header, here is one sample Library you can try for .NET which creates a HTTP Request with signed header.
If your clients cannot send Authorization Header or cannot use above library then you can convert the signature to be a Query String Format and provide the pre-signed Urls to them.
This AWS Documentation has example in Python on how to generate Query String URL. Now, you can take python example and convert into .NET based code with following sample.
public string GetSig4QueryString(string host, string service, string region)
{
var t = DateTimeOffset.UtcNow;
var amzdate = t.ToString("yyyyMMddTHHmmssZ");
var datestamp = t.ToString("yyyyMMdd");
var canonical_uri = "/dev/myApigNodeJS";
var canonical_headers = "host:" + host+"\n";
var signed_headers = "host";
var credential_scope = $"{datestamp}/{region}/{service}/aws4_request";
var canonical_querystring = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=" + WebUtility.UrlEncode(_access_key + "/" + credential_scope)
+ "&X-Amz-Date=" + amzdate + "&X-Amz-SignedHeaders=" + signed_headers;
Console.WriteLine("canonical_querystring");
Console.WriteLine(canonical_querystring);
var payload_hash = Hash(new byte[0]);//No Payload for GET
var canonical_request = new StringBuilder();
canonical_request.Append("GET\n");
canonical_request.Append(canonical_uri + "\n");
canonical_request.Append(canonical_querystring + "\n");
canonical_request.Append(canonical_headers + "\n");
canonical_request.Append(signed_headers + "\n");
canonical_request.Append(payload_hash);
Console.WriteLine("canonical_request");
Console.WriteLine(canonical_request);
var string_to_sign = $"{algorithm}\n{amzdate}\n{credential_scope}\n" + Hash(Encoding.UTF8.GetBytes(canonical_request.ToString()));
Console.WriteLine("string_to_sign");
Console.WriteLine(string_to_sign);
var signing_key = GetSignatureKey(_secret_key, datestamp, region, service);
var signature = ToHexString(HmacSHA256(signing_key, string_to_sign));
var signed_querystring = canonical_querystring+"&X-Amz-Signature=" + signature;
return signed_querystring;
}
GetSig4QueryString("myApiId.execute-api.us-east-1.amazonaws.com","execute-api","us-east-1");
//Returned String --> X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Full Endpoint Becomes -
https://myApiId.execute-api.us-east-1.amazonaws.com/dev/myApigNodeJS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Note -
This example code refers methods & variables from Github project I gave above.
Also, this example hard coded API Path /dev/myApigNodeJS and signs it and it will be different for you with full absolute path.
AWS recommends to sign all queryStrings, headers which you are planning to send in request. Go through .NET code of library I referred and understand how its doing that.
Let me know if you have questions.
When generating the presigned url a websocket service within the AWS API Gateway, I used the solution by Imran and added the "X-Amz-Security-Token" which is required.
Related
I encounter a SignNotMatch when I generate a presigned url with boto3 with the code below:
session = Session(access_key, secret_key)
s3 = session.client('s3', endpoint_url=OSS_ENDPOINT, config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url(
ClientMethod='get_object',
Params={
'Bucket': bucket,
'Key': key
}
)
And then parse the request and resign(use AWS Go SDK signer.Presign) it in our proxy, and I always get a not match error.
Then I open the debug mode in boto3 and add log in AWS Go SDK, and found that when they calculate Canonical Request they use different way:
Canonical Request:
HTTP Verb + '\n' +
Canonical URI + '\n' +
Canonical Query String + '\n' +
Signed Headers + '\n' +
"UNSIGNED-PAYLOAD"
In Go AWS SDK it will put X-Amz-Content-Sha256=UNSIGNED-PAYLOADin Canonical Query String by default while boto3 will not.
Is it supposed to or I use it in a wrong way?
I create a pre-signed URL and get back something like
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
I can now curl this no problem. However, if I now add another query parameter, I will get back a 403, i.e.
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
&Foo=123
How come? Is it possible to generate a pre-signed url that supports custom queries?
It seems to be technically feasible to insert custom query parameters into a v4 pre-signed URL, before it is signed, but not all of the AWS SDKs expose a way to do this.
Here's an example of a roundabout way to do this with the AWS JavaScript SDK:
const AWS = require('aws-sdk');
var s3 = new AWS.S3({region: 'us-east-1', signatureVersion: 'v4'});
var req = s3.getObject({Bucket: 'mybucket', Key: 'mykey'});
req.on('build', () => { req.httpRequest.path += '?session=ABC123'; });
console.log(req.presign());
I've tried this with custom query parameters that begin with X- and without it. Both appeared to work fine. I've tried with multiple query parameters (?a=1&b=2) and that worked too.
The customized pre-signed URLs work correctly (I can use them to get S3 objects) and the query parameters make it into CloudWatch Logs so can be used for correlation purposes.
Note that if you want to supply a custom expiration time, then do it as follows:
const Expires = 120;
const url = req.presign(Expires);
I'm not aware of other (non-JavaScript) SDKs that allow you to insert query parameters into the URL construction process like this so it may be a challenge to do this in other languages. I'd recommend using a small JavaScript Lambda function (or API Gateway plus Lambda function) that would simply create and return the customized pre-signed URL.
The custom query parameters are also tamper-proof. They are included in the signing of the URL so, if you tamper with them, the URL becomes invalid, yielding 403 Forbidden.
I used this code to generate your pre-signed URL. The result was:
https://s3.amazonaws.com/MyBucket/MyItem
?Foo=123
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA...27%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20180427T0012345Z
&X-Amz-Expires=3600
&X-Amz-Signature=e3...7b
&X-Amz-SignedHeaders=host
None of this is a guarantee that this technique will continue to work, of course, if AWS changes things under the covers but for right now it seems to work and is certainly useful.
Attribution: the source of this discovery was aws-sdk-js/issues/502.
If you change one of the headers or add / subtract, then you have to resign the URL.
This is part of the AWS signing design and this process is designed for higher levels of security. One of the AWS reasons for changing to signing version 4 from signing version 2.
The signing design does not know which headers are important and which are not. That would create a nightmare trying to track all of the AWS services.
I created this solution for Ruby SDK. It is sort of a hack, but it works as expected:
require 'aws-sdk-s3'
require 'active_support/core_ext/object/to_query.rb'
# Modified S3 pre signer class that can inject query params to the URL
#
# Usage example:
#
# bucket_name = "bucket_name"
# key = "path/to/file.json"
# filename = "download_file_name.json"
# duration = 3600
#
# params = {
# bucket: bucket_name,
# key: key,
# response_content_disposition: "attachment; filename=#{filename}",
# expires_in: duration
# }
#
# signer = S3PreSignerWithQueryParams.new({'x-your-custom-field': "banana", 'x-some-other-field': 1234})
# url = signer.presigned_url(:get_object, params)
#
# puts "url = #{url}"
#
class S3PreSignerWithQueryParams < Aws::S3::Presigner
def initialize(query_params = {}, options = {})
#query_params = query_params
super(options)
end
def build_signer(cfg)
signer = super(cfg)
my_params = #query_params.to_h.to_query()
signer.define_singleton_method(:presign_url,
lambda do |options|
options[:url].query += "&" + my_params
super(options)
end)
signer
end
end
While not documented, you can add parameters as arguments to the call to presigned_url.
obj.presigned_url(:get,
expires_in: expires_in_sec,
response_content_disposition: "attachment"
)
https://bucket.s3.us-east-2.amazonaws.com/file.txt?response-content-disposition=attachment&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=PUBLICKEY%2F20220309%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220309T031958Z&X-Amz-Expires=43200&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATUREVALUE
If you are looking on for JavaScript SDK V3:
import { HttpRequest } from "#aws-sdk/protocol-http";
import { S3RequestPresigner } from "#aws-sdk/s3-request-presigner";
import { parseUrl } from "#aws-sdk/url-parser";
import { Sha256 } from "#aws-crypto/sha256-browser";
import { Hash } from "#aws-sdk/hash-node";
import { formatUrl } from "#aws-sdk/util-format-url";
// Make custom query in Record<string, string | Array<string> | null> format
const customQuery = {
hello: "world",
};
const s3ObjectUrl = parseUrl(
`https://${bucketName}.s3.${region}.amazonaws.com/${key}`
);
s3ObjectUrl.query = customQuery; //Insert custom query here
const presigner = new S3RequestPresigner({
credentials,
region,
sha256: Hash.bind(null, "sha256"), // In Node.js
//sha256: Sha256 // In browsers
});
// Create a GET request from S3 url.
const url = await presigner.presign(new HttpRequest(s3ObjectUrl));
console.log("PRESIGNED URL: ", formatUrl(url));
Code template taken from: https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/
I am developing an application that needs to send files to Google Cloud Storage.
The webapp will have a HTML page that the user choose files to do upload.
The user do not have Google Account.
The amount files to send is 5 or less.
I do not want to send files to GAE and GAE send to GCS. I would like that my user to do upload directly to GCS.
I did this code for upload:
function sentStorage() {
var file = document.getElementById("myFile").files[0];
url = 'https://www.googleapis.com/upload/storage/v1/b/XXX/o?uploadType=resumable&name=' + file.name;
xhr = new XMLHttpRequest();
var token = 'ya29.XXXXXXXXXXXXXXX';
xhr.open('POST', url);
xhr.setRequestHeader('Content-Type', file.type);
// resumable
//url = 'https://www.googleapis.com/upload/storage/v1/b/XXXXXX/o?uploadType=resumable&name=' + file.name;
//xhr.setRequestHeader('Content-Type', 'application/json; charset=UTF-8');
//xhr.setRequestHeader('Content-Length', file.size);
xhr.setRequestHeader('x-goog-project-id', 'XXXXXXXXXX');
xhr.setRequestHeader('Authorization', 'Bearer ' + token);
xhr.send(file);
xhr.onreadystatechange = function () {
if (xhr.readyState === 4) {
var response = JSON.parse(xhr.responseText);
if (xhr.status === 200) {
alert('codigo 200');
} else {
var message = 'Error: ' + response.error.message;
console.log(message);
alert(message);
}
}
};
}
I get a serviceaccount information (Google Console) and generate a token Bearer for it. I used a python file that read the "json account information" and generate the token.
My requisit is that user do not need to confirm any Google Account information for send files, this obligation is from my application. (Users do not have Google Account) and the html page send the files directly to GCS without send to GAE or GCE, so, I need to use HTML form or Javascript. I prefer Javascript.
Only users of this application can do upload (the application has an authentication with database), so, anonymous user can not do it.
My questions are:
This token will expire? I used a serviceaccount for generate this token.
There is a better api javascript to do it?
This security solution is better or I should use a different approach?
Sending either a refresh or an access token to an untrusted end user is very dangerous. The bearer of an access token has complete authority to act as the associated account (within the scope used to generate it) until the access token expires a few minutes later. You don't want to do that.
There are a few good alternatives. The easiest way is to create exactly the upload request you want, then sign the URL for that request using the private key of a service account. That signed URL, which will be valid for a few minutes, could then be used to upload a single object. You'll need to sign the URL on the server side before giving it to the customer. Here's the documentation on signed URLs: https://cloud.google.com/storage/docs/access-control/signed-urls
Cloudfront supports signed cookies for serving up private content but I cant find any examples on how to do this.
I have found examples on how to sign URLs with the Java AWS API but not Cookies, can someone please share their experiences with doing this and is this the best way to secure multiple forms of media being served from CloudFront.
Our site has images and video that are uploaded by the user, which can then be viewed by searches on our site, I want to make sure that these images can only be served by our site and not copied for later use.
We were able to introduce signed cookies with custom policies using
this library
http://www.jets3t.org/
You need three cookies created by your app as described here
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-setting-signed-cookie-custom-policy.html
Please read that carefully. Especially the part on how to create a policy.
The three cookies are:
CloudFront-Policy
CloudFront-Signature
CloudFront-Key-Pair-Id
First create a policy
Date expirationTime = (new LocalDate()).plusYears(1).toDate();
String customPolicy = CloudFrontService.buildPolicyForSignedUrl(basePath, expirationTime, null, null);
//and assign it to a cookie
Cookie signedCookiePolicy = new Cookie("CloudFront-Policy", ServiceUtils.toBase64(customPolicy.getBytes()));
signedCookiePolicy.setMaxAge(365 * 24 * 60 * 60);
signedCookiePolicy.setPath("/");
response.addCookie(signedCookiePolicy);
The signature is the tricky part but all tools are available once you use this jets3t thing
byte[] signatureBytes = EncryptionUtil.signWithRsaSha1(getDerPrivateKey(), customPolicy.getBytes("UTF-8"));
String signature = ServiceUtils.toBase64(signatureBytes).replace('+', '-').replace('=', '_').replace('/', '~');
Cookie signedCookieSignagture = new Cookie("CloudFront-Signature",cdnSignService.signBaseUrl(basePath, expirationTime));
signedCookieSignagture.setMaxAge(365 * 24 * 60 * 60);
signedCookieSignagture.setPath("/");
response.addCookie(signedCookieSignagture);
The third cookie only holds the key-id of your AWS account.
Cookie signedCookieKeyPairId = new Cookie("CloudFront-Key-Pair-Id","YOUR_AWS_CF_KEY_ID");
signedCookieKeyPairId.setMaxAge(365 * 24 * 60 * 60);
signedCookieKeyPairId.setPath("/");
response.addCookie(signedCookieKeyPairId);
The above only introduces you to concepts of using the correct libs to create the signed cookies. Its not executable or runnable on its own.
Be nice, its my first overflow contribution..
In AWS JAVA SDK version 1.10.73 introduced class CloudFrontCookieSigner for Signed Cookies with custom policies. Using this class and methods we can generate cookies.
CloudFront-Signature
CloudFront-Policy
CloudFront-Key-Pair-Id
Note that Java only supports SSL certificates in DER format,so you will need to convert your PEM-formatted file to DER format.
To do this, you can use openssl:
Command to Generate .der File from .pem
openssl pkcs8 -topk8 -nocrypt -in origin.pem -inform PEM -out new.der -outform DER
Reference:- http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CFPrivateDistJavaDevelopment.html
String privateKeyFilecdr = "/home/ec2-user/cookie.der";
String distributionDomain = "xxxxxxxx.cloudfront.net";
String s3ObjectKey = "signed-cookie.png";
String distributionid = "X4X4X4Y5Y5"; //Cloud Front Distribution id
String KeyFileId = "MPSIKFGHLMNSTOP" //AWS PEM KEY FILE ID
Date expiresOn = DateUtils.parseISO8601Date("2012-11-14T22:20:00.000Z");
String policyResourcePath = "https://" + distributionDomain + "/" + s3ObjectKey;
File privateKeyFile = new File(privateKeyFilecdr);
CookiesForCannedPolicy cookies = null;
try {
cookies = CloudFrontCookieSigner.getCookiesForCannedPolicy(policyResourcePath, KeyFileId, privateKeyFile, expiresOn);
// #SuppressWarnings({ "resource", "deprecation" })
HttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(SignerUtils.generateResourcePath(Protocol.https, distributionDomain,
s3ObjectKey));
httpGet.addHeader("Cookie", cookies.getExpires().getKey() + "=" +
cookies.getExpires().getValue());
httpGet.addHeader("Cookie", cookies.getSignature().getKey() + "=" +
cookies.getSignature().getValue());
httpGet.addHeader("Cookie", cookies.getKeyPairId().getKey() + "=" +
cookies.getKeyPairId().getValue());
HttpResponse responsevalues = client.execute(httpGet);
// System.out.println(responsevalues);
} catch (InvalidKeySpecException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I'm familiar with the Range HTTP header; however, the interface I'm using to query S3 (an img element's .src property) doesn't allow me to specify HTTP headers.
Is there a way for me to specify my desired range via a parameter in the query string?
It doesn't seem like there is, but I'm just holding out a shred of hope before I roll my own solution with ajax requests.
Amazon S3 supports Range GET requests, as do some HTTP servers, for example, Apache and IIS.
How CloudFront Processes Partial Requests for an Object (Range GETs)
I tried to get my S3 object via cURL:
curl -r 0-1024 https://s3.amazonaws.com/mybucket/myobject -o part1
curl -r 1025- https://s3.amazonaws.com/mybucket/myobject -o part2
cat part1 part2 > myobject
and AWS SDK for JavaScript:
var s3 = new AWS.S3();
var file = require('fs').createWriteStream('part1');
var params = {
Bucket: 'mybucket',
Key: 'myobject',
Range: 'bytes=0-1024'
};
s3.getObject(params).createReadStream().pipe(file);
These two methods work fine for me.
AWS SDK for JavaScript API Reference (getObject)
Below is the Java Code with AWS V2 SDK
format the range as below
var range = String.format("bytes=%d-%d", start, end);
and pass it in below api with GetObjectRequest builder
ResponseBytes<GetObjectResponse> currentS3Obj = client.getObjectAsBytes(GetObjectRequest.builder().bucket(bucket).key(key).range(range).build());
return currentS3Obj.asInputStream();