I'm familiar with the Range HTTP header; however, the interface I'm using to query S3 (an img element's .src property) doesn't allow me to specify HTTP headers.
Is there a way for me to specify my desired range via a parameter in the query string?
It doesn't seem like there is, but I'm just holding out a shred of hope before I roll my own solution with ajax requests.
Amazon S3 supports Range GET requests, as do some HTTP servers, for example, Apache and IIS.
How CloudFront Processes Partial Requests for an Object (Range GETs)
I tried to get my S3 object via cURL:
curl -r 0-1024 https://s3.amazonaws.com/mybucket/myobject -o part1
curl -r 1025- https://s3.amazonaws.com/mybucket/myobject -o part2
cat part1 part2 > myobject
and AWS SDK for JavaScript:
var s3 = new AWS.S3();
var file = require('fs').createWriteStream('part1');
var params = {
Bucket: 'mybucket',
Key: 'myobject',
Range: 'bytes=0-1024'
};
s3.getObject(params).createReadStream().pipe(file);
These two methods work fine for me.
AWS SDK for JavaScript API Reference (getObject)
Below is the Java Code with AWS V2 SDK
format the range as below
var range = String.format("bytes=%d-%d", start, end);
and pass it in below api with GetObjectRequest builder
ResponseBytes<GetObjectResponse> currentS3Obj = client.getObjectAsBytes(GetObjectRequest.builder().bucket(bucket).key(key).range(range).build());
return currentS3Obj.asInputStream();
Related
Task: Upload an already-compressed file to an Amazon AWS S3 bucket using the AWSSDK.S3 package in .NET Core, and set the Content-Encoding header to "gzip".
I have an S3 bucket that I am trying to upload to. I'm using an IO stream for the data, but there's no way to set the item's Content-Encoding during the upload process. This is the header I want to set that the item will have when served up by the S3 service (it is not the Content-Encoding for the upload itself).
I am using the TransferUtility and TransferUtilityUploadRequest objects. Whereas you can set Content-Type just fine (see below), there is no property to set Content-Encoding. If you use the Metadata.Add function, it will automatically rewrite Content-Encoding to a custom "x-amz-meta-content-encoding" key, which is of course useless.
From what I understand you can't set the Metadata in S3 after the fact except by copying the object, or doing it yourself manually (this is a no-go for me, too many files). With copying the object, I'm not sure I'd be able to set the metadata anyway.
var s3Client = new AmazonS3Client(awsCredentials, bucketRegion);
var fileTransferUtility = new TransferUtility(s3Client);
byte[] compressedDataArray = <some compressed data>;
using (MemoryStream memoryStream = new MemoryStream(compressedDataArray)) {
var fileTransferUtilityRequest = new TransferUtilityUploadRequest {
BucketName = bucketName,
ContentType = "application/json",
StorageClass = S3StorageClass.Standard,
Key = fileKey + ".gz",
CannedACL = S3CannedACL.PublicRead,
InputStream = memoryStream,
AutoCloseStream = true
};
fileTransferUtilityRequest.Metadata.Add("Content-Encoding", "gzip");
fileTransferUtility.Upload(fileTransferUtilityRequest);
}
It will upload to a the new file in S3, but will not have Content-Encoding header set.
Please help, anyone! thanks!
okay what seems to work is to set the ContentEncoding on the Headers collection for the request:
fileTransferUtilityRequest.Headers.ContentEncoding = "gzip";
I am generating a presigned URL server-side to allow my client application to upload a file directly to the S3 bucket. Everything works fine unless the client application is running on a computer in a timezone that is technically a day ahead of my server clock.
I can recreate the issue locally by setting my system clock ahead to a timezone on the next day.
Here is how I am generating the presigned URL using the .NET SDK (I originally had DateTime.Now instead of UTCNow):
var request = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = objectName,
Verb = HttpVerb.PUT,
Expires = DateTime.UtcNow.AddDays(5),
ContentType = "application/octet-stream"
};
request.Headers["x-amz-acl"] = "bucket-owner-full-control";
request.Metadata.Add("call", JsonConvert.SerializeObject(call).ToString());
return client.GetPreSignedURL(request);
and then I am using that presigned URL in the client application like this:
using (var fileStream = new FileStream(recordingPath, FileMode.Open))
using (var client = new WebClient())
{
HttpContent fileStreamContent = new StreamContent(fileStream);
var bytes = await fileStreamContent.ReadAsByteArrayAsync();
client.Headers.Add("Content-Type", "application/octet-stream");
//include metadata in PUT request
client.Headers.Add("x-amz-meta-call", JsonConvert.SerializeObject(Call));
await client.UploadDataTaskAsync(new Uri(presignedUrl), "PUT", bytes);
}
Here is the error I am receiving from AWS:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>{access}</AWSAccessKeyId><StringToSign>....
The requests appear mostly identical to me in Fiddler.
Works:
PUT https://{bucketname}.s3.amazonaws.com/1c849c76-dd2a-4ff7-aad7-23ec7e9ddd45_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={cred}&X-Amz-Date=20190312T021419Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 28289
Expect: 100-continue
{file}
Does not work:
PUT https://{bucketname}.s3.amazonaws.com/4cca3ec3-9f3f-4ba4-9d81-6336090610c0_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={credentials}&X-Amz-Date=20190312T021541Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 18714
Expect: 100-continue
{file}
In both scenarios, the presigned URL has the same x-amz-date parameter generated. I have even tried parsing out the x-amz-date parameter from the URL and explicitly setting it as a header in my PUT but that did not work either.
What am I missing?
It turned out to me that I was using a different version of the signature.
v4 worked perfectly for me.
In JS, require S3 as
const s3 = new AWS.S3({
signatureVersion: 'v4'
});
So the issue ended up being within the metadata. In our setup, we had the client application posting a JSON string up to our API along with the file to generate the presigned URL. We were using Json.net to deserialize into the C# class:
var call = JsonConvert.DeserializeObject<Call>(request.Params["metadata"]);
Apparently, this call converts any timestamps in the Json to local time. This means that we would sign the URL with metadata timestamps local to the API server, but actually upload the file with metadata timestamps local to the client. This difference is why the calculated signatures are different.
I have seen pre-signed URL for S3 object. Is it possible to create pre-signed URL for API gateway. I have gone through documentation. I am using .NET. I would like to know if there is .NET library available to create pre-signed request for gateway API.
ISSUE
I have GET API something like this https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/pets?type=dog&page=1 and our client is going to invoke that API once in a while. The legacy tool that they are using only supports GET. So i wanted to create a pre-signed URL (with short expiry time) and give them when they ask for it. For each client i already have IAM user with their respective accesskey and secretkey
PreSigned URLs are typically signed with AWS SigV4 signing process.
You can generate SigV4 signed Urls for your API Gateway Hosted Endpoints. Typically, you will need to send SigV4 signature in Authorization Request Header. If you are clients are willing to send header, here is one sample Library you can try for .NET which creates a HTTP Request with signed header.
If your clients cannot send Authorization Header or cannot use above library then you can convert the signature to be a Query String Format and provide the pre-signed Urls to them.
This AWS Documentation has example in Python on how to generate Query String URL. Now, you can take python example and convert into .NET based code with following sample.
public string GetSig4QueryString(string host, string service, string region)
{
var t = DateTimeOffset.UtcNow;
var amzdate = t.ToString("yyyyMMddTHHmmssZ");
var datestamp = t.ToString("yyyyMMdd");
var canonical_uri = "/dev/myApigNodeJS";
var canonical_headers = "host:" + host+"\n";
var signed_headers = "host";
var credential_scope = $"{datestamp}/{region}/{service}/aws4_request";
var canonical_querystring = "X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=" + WebUtility.UrlEncode(_access_key + "/" + credential_scope)
+ "&X-Amz-Date=" + amzdate + "&X-Amz-SignedHeaders=" + signed_headers;
Console.WriteLine("canonical_querystring");
Console.WriteLine(canonical_querystring);
var payload_hash = Hash(new byte[0]);//No Payload for GET
var canonical_request = new StringBuilder();
canonical_request.Append("GET\n");
canonical_request.Append(canonical_uri + "\n");
canonical_request.Append(canonical_querystring + "\n");
canonical_request.Append(canonical_headers + "\n");
canonical_request.Append(signed_headers + "\n");
canonical_request.Append(payload_hash);
Console.WriteLine("canonical_request");
Console.WriteLine(canonical_request);
var string_to_sign = $"{algorithm}\n{amzdate}\n{credential_scope}\n" + Hash(Encoding.UTF8.GetBytes(canonical_request.ToString()));
Console.WriteLine("string_to_sign");
Console.WriteLine(string_to_sign);
var signing_key = GetSignatureKey(_secret_key, datestamp, region, service);
var signature = ToHexString(HmacSHA256(signing_key, string_to_sign));
var signed_querystring = canonical_querystring+"&X-Amz-Signature=" + signature;
return signed_querystring;
}
GetSig4QueryString("myApiId.execute-api.us-east-1.amazonaws.com","execute-api","us-east-1");
//Returned String --> X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Full Endpoint Becomes -
https://myApiId.execute-api.us-east-1.amazonaws.com/dev/myApigNodeJS?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20190104%2Fus-east-1%2Fexecute-api%2Faws4_request&X-Amz-Date=20190104T190309Z&X-Amz-SignedHeaders=host&X-Amz-Signature=7b830fce28f7800b3879a25850950f6c4247dfdc07775b6952295fa2fff03f7f
Note -
This example code refers methods & variables from Github project I gave above.
Also, this example hard coded API Path /dev/myApigNodeJS and signs it and it will be different for you with full absolute path.
AWS recommends to sign all queryStrings, headers which you are planning to send in request. Go through .NET code of library I referred and understand how its doing that.
Let me know if you have questions.
When generating the presigned url a websocket service within the AWS API Gateway, I used the solution by Imran and added the "X-Amz-Security-Token" which is required.
I create a pre-signed URL and get back something like
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
I can now curl this no problem. However, if I now add another query parameter, I will get back a 403, i.e.
https://s3.amazonaws.com/MyBucket/MyItem/
?X-Amz-Security-Token=TOKEN
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Date=20171206T014837Z
&X-Amz-SignedHeaders=host
&X-Amz-Expires=3600
&X-Amz-Credential=CREDENTIAL
&X-Amz-Signature=SIGNATURE
&Foo=123
How come? Is it possible to generate a pre-signed url that supports custom queries?
It seems to be technically feasible to insert custom query parameters into a v4 pre-signed URL, before it is signed, but not all of the AWS SDKs expose a way to do this.
Here's an example of a roundabout way to do this with the AWS JavaScript SDK:
const AWS = require('aws-sdk');
var s3 = new AWS.S3({region: 'us-east-1', signatureVersion: 'v4'});
var req = s3.getObject({Bucket: 'mybucket', Key: 'mykey'});
req.on('build', () => { req.httpRequest.path += '?session=ABC123'; });
console.log(req.presign());
I've tried this with custom query parameters that begin with X- and without it. Both appeared to work fine. I've tried with multiple query parameters (?a=1&b=2) and that worked too.
The customized pre-signed URLs work correctly (I can use them to get S3 objects) and the query parameters make it into CloudWatch Logs so can be used for correlation purposes.
Note that if you want to supply a custom expiration time, then do it as follows:
const Expires = 120;
const url = req.presign(Expires);
I'm not aware of other (non-JavaScript) SDKs that allow you to insert query parameters into the URL construction process like this so it may be a challenge to do this in other languages. I'd recommend using a small JavaScript Lambda function (or API Gateway plus Lambda function) that would simply create and return the customized pre-signed URL.
The custom query parameters are also tamper-proof. They are included in the signing of the URL so, if you tamper with them, the URL becomes invalid, yielding 403 Forbidden.
I used this code to generate your pre-signed URL. The result was:
https://s3.amazonaws.com/MyBucket/MyItem
?Foo=123
&X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=AKIA...27%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20180427T0012345Z
&X-Amz-Expires=3600
&X-Amz-Signature=e3...7b
&X-Amz-SignedHeaders=host
None of this is a guarantee that this technique will continue to work, of course, if AWS changes things under the covers but for right now it seems to work and is certainly useful.
Attribution: the source of this discovery was aws-sdk-js/issues/502.
If you change one of the headers or add / subtract, then you have to resign the URL.
This is part of the AWS signing design and this process is designed for higher levels of security. One of the AWS reasons for changing to signing version 4 from signing version 2.
The signing design does not know which headers are important and which are not. That would create a nightmare trying to track all of the AWS services.
I created this solution for Ruby SDK. It is sort of a hack, but it works as expected:
require 'aws-sdk-s3'
require 'active_support/core_ext/object/to_query.rb'
# Modified S3 pre signer class that can inject query params to the URL
#
# Usage example:
#
# bucket_name = "bucket_name"
# key = "path/to/file.json"
# filename = "download_file_name.json"
# duration = 3600
#
# params = {
# bucket: bucket_name,
# key: key,
# response_content_disposition: "attachment; filename=#{filename}",
# expires_in: duration
# }
#
# signer = S3PreSignerWithQueryParams.new({'x-your-custom-field': "banana", 'x-some-other-field': 1234})
# url = signer.presigned_url(:get_object, params)
#
# puts "url = #{url}"
#
class S3PreSignerWithQueryParams < Aws::S3::Presigner
def initialize(query_params = {}, options = {})
#query_params = query_params
super(options)
end
def build_signer(cfg)
signer = super(cfg)
my_params = #query_params.to_h.to_query()
signer.define_singleton_method(:presign_url,
lambda do |options|
options[:url].query += "&" + my_params
super(options)
end)
signer
end
end
While not documented, you can add parameters as arguments to the call to presigned_url.
obj.presigned_url(:get,
expires_in: expires_in_sec,
response_content_disposition: "attachment"
)
https://bucket.s3.us-east-2.amazonaws.com/file.txt?response-content-disposition=attachment&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=PUBLICKEY%2F20220309%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20220309T031958Z&X-Amz-Expires=43200&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATUREVALUE
If you are looking on for JavaScript SDK V3:
import { HttpRequest } from "#aws-sdk/protocol-http";
import { S3RequestPresigner } from "#aws-sdk/s3-request-presigner";
import { parseUrl } from "#aws-sdk/url-parser";
import { Sha256 } from "#aws-crypto/sha256-browser";
import { Hash } from "#aws-sdk/hash-node";
import { formatUrl } from "#aws-sdk/util-format-url";
// Make custom query in Record<string, string | Array<string> | null> format
const customQuery = {
hello: "world",
};
const s3ObjectUrl = parseUrl(
`https://${bucketName}.s3.${region}.amazonaws.com/${key}`
);
s3ObjectUrl.query = customQuery; //Insert custom query here
const presigner = new S3RequestPresigner({
credentials,
region,
sha256: Hash.bind(null, "sha256"), // In Node.js
//sha256: Sha256 // In browsers
});
// Create a GET request from S3 url.
const url = await presigner.presign(new HttpRequest(s3ObjectUrl));
console.log("PRESIGNED URL: ", formatUrl(url));
Code template taken from: https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/
I'm trying to upload an image to S3 via putObject and a pre-signed URL.
Here is the URL that was provided when I generated the pre-signed URL:
https://<myS3Bucket>.s3.amazonaws.com/1ffd1c88-5661-48f9-a135-04bd569614dd.jpg?AWSAccessKeyId=<accessKey>&Expires=1458177431311&Signature=<signature>-amz-security-token=<token>
When I attempt to upload the file via a PUT, AWS responds with:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Invalid date (should be seconds since epoch): 1458177431311</Message>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>
Here is the curl version of the request I was using:
curl -X PUT -H "Cache-Control: no-cache" -H "Postman-Token: 78e46be3-8ecc- 4156-be3d-7e2f4688a127" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "file=#[object Object]" "https://<myS3Bucket>.s3.amazonaws.com/1ffd1c88-5661-48f9-a135-04bd569614dd.jpg?AWSAccessKeyId=<accessKey>&Expires=1458177431311&Signature=<signature>-amz-security-token=<mySecurityToken>"
Since the timestamp is generated by AWS, it should be correct. I have tried changing it to include decimals and got the same error.
Could the problem be in the way I'm uploading the file in my request?
Update - Add code for generating the signed URL
The signed URL is being generated via the AWS Javascript SDK:
var AWS = require('aws-sdk')
var uuid = require('node-uuid')
var Promise = require('bluebird')
var s3 = new AWS.S3()
var params = {
Bucket: bucket, // bucket is stored as .env variable
Key: uuid.v4() + '.jpg' // file is always a jpg
}
return new Promise(function (resolve, reject) {
s3.getSignedUrl('putObject', params, function (err, url) {
if (err) {
reject(new Error(err))
}
var payload = { url: url }
resolve(payload)
})
})
My access key and secret key are loaded via environment variables as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Amazon S3 only supports 32 bit timestamps in signed urls:
2147483647 is the largest timestamp you can have.
I was creating timestamps 20 years in the future and it was breaking my signed URL's. Using a value less than 2,147,483,647 fixes the issue.
Hope this helps someone!
You just get bitten by bad documentation. Seems relate to this link
Amazon S3 invalid date when using expires in url_for
The integer are mean for " (to specify the number of seconds after the current time)".
Since you enter the time in epoch(0 = 1970-1-1), So it is current epoch time + your time, 46+46 years= 92 years. Appear to be crazy expiration period for s3.