SignatureDoesNotMatch on S3 PUT Request to Presigned URL - amazon-web-services

I am generating a presigned URL server-side to allow my client application to upload a file directly to the S3 bucket. Everything works fine unless the client application is running on a computer in a timezone that is technically a day ahead of my server clock.
I can recreate the issue locally by setting my system clock ahead to a timezone on the next day.
Here is how I am generating the presigned URL using the .NET SDK (I originally had DateTime.Now instead of UTCNow):
var request = new GetPreSignedUrlRequest
{
BucketName = bucketName,
Key = objectName,
Verb = HttpVerb.PUT,
Expires = DateTime.UtcNow.AddDays(5),
ContentType = "application/octet-stream"
};
request.Headers["x-amz-acl"] = "bucket-owner-full-control";
request.Metadata.Add("call", JsonConvert.SerializeObject(call).ToString());
return client.GetPreSignedURL(request);
and then I am using that presigned URL in the client application like this:
using (var fileStream = new FileStream(recordingPath, FileMode.Open))
using (var client = new WebClient())
{
HttpContent fileStreamContent = new StreamContent(fileStream);
var bytes = await fileStreamContent.ReadAsByteArrayAsync();
client.Headers.Add("Content-Type", "application/octet-stream");
//include metadata in PUT request
client.Headers.Add("x-amz-meta-call", JsonConvert.SerializeObject(Call));
await client.UploadDataTaskAsync(new Uri(presignedUrl), "PUT", bytes);
}
Here is the error I am receiving from AWS:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>{access}</AWSAccessKeyId><StringToSign>....
The requests appear mostly identical to me in Fiddler.
Works:
PUT https://{bucketname}.s3.amazonaws.com/1c849c76-dd2a-4ff7-aad7-23ec7e9ddd45_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={cred}&X-Amz-Date=20190312T021419Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 28289
Expect: 100-continue
{file}
Does not work:
PUT https://{bucketname}.s3.amazonaws.com/4cca3ec3-9f3f-4ba4-9d81-6336090610c0_encoded.opus?X-Amz-Expires=18000&x-amz-security-token={security_token}&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential={credentials}&X-Amz-Date=20190312T021541Z&X-Amz-SignedHeaders=content-type;host;x-amz-acl;x-amz-meta-call;x-amz-security-token&X-Amz-Signature={sig} HTTP/1.1
x-amz-meta-call: {json_string}
x-amz-acl: bucket-owner-full-control
Content-Type: application/octet-stream
Host: {bucketname}.s3.amazonaws.com
Content-Length: 18714
Expect: 100-continue
{file}
In both scenarios, the presigned URL has the same x-amz-date parameter generated. I have even tried parsing out the x-amz-date parameter from the URL and explicitly setting it as a header in my PUT but that did not work either.
What am I missing?

It turned out to me that I was using a different version of the signature.
v4 worked perfectly for me.
In JS, require S3 as
const s3 = new AWS.S3({
signatureVersion: 'v4'
});

So the issue ended up being within the metadata. In our setup, we had the client application posting a JSON string up to our API along with the file to generate the presigned URL. We were using Json.net to deserialize into the C# class:
var call = JsonConvert.DeserializeObject<Call>(request.Params["metadata"]);
Apparently, this call converts any timestamps in the Json to local time. This means that we would sign the URL with metadata timestamps local to the API server, but actually upload the file with metadata timestamps local to the client. This difference is why the calculated signatures are different.

Related

Uploading Base64 file to S3 signed URL

I need to upload an image to S3 using signed URL. I have the image in a base64 string. The below code runs without throwing any error, but at the end I see a text file with base64 content in the S3, not the binary image.
Can you please point out what I am missing?
Generate Signed URL (Lambda function JavaScript)
const signedUrlExpireSeconds = 60 * 100;
var url = s3.getSignedUrl("putObject", {
Bucket: process.env.ScreenshotBucket,
Key: s3Key,
ContentType: "image/jpeg",
ContentEncoding: "base64",
Expires: signedUrlExpireSeconds,
});
Upload to S3 (Java Code)
HttpRequest request = HttpRequest.newBuilder().PUT(HttpRequest.BodyPublishers.ofString(body))
.header("Content-Encoding", "base64").header("Content-Type", "image/jpeg").uri(URI.create(url)).build();
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() != 200) {
throw new Exception(response.body());
}
I am not familiar with the AWS JavaScript SDK. But it seems that setting the 'Content-Type' metadata of the object (not the Content-Type of the putObject HTTP request) to 'image/jpeg' should do the trick.
Fixed it while just playing around with the combinations.
HttpRequest request = HttpRequest.newBuilder().PUT(HttpRequest.BodyPublishers.ofString(body))
Changed to
HttpRequest request = HttpRequest.newBuilder().PUT(HttpRequest.BodyPublishers.ofByteArray(body))

Call S3 pre-signed URL with postman

I am attempting to use a pre-signed URL to upload as described in the docs (https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html) I can retrieve the pre-signed URL but when I attempt to do a PUT in Postman, I receive the following error:
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
Obviously, the way my put call is structured doesn't match with the way AWS is calculating the signature. I can't find a lot of information on what this put call requires.
I've attempted to modify the header for Content-Type to multipart/form-data and application/octet-stream. I've also tried to untick the headers section in postman and rely on the body type for both form-data and binary settings where I select the file. The form-data setting results in the following added to the call:
Content-Disposition: form-data; name="thefiletosend.txt"; filename="thefiletosend.txt
In addition, I noticed that postman is including what it calls "temporary headers" as follows:
Host: s3.amazonaws.com
Content-Type: text/plain
User-Agent: PostmanRuntime/7.13.0
Accept: /
Cache-Control: no-cache
Postman-Token: e11d1ef0-8156-4ca7-9317-9f4d22daf6c5,2135bc0e-1285-4438-bb8e-b21d31dc36db
Host: s3.amazonaws.com
accept-encoding: gzip, deflate
content-length: 14
Connection: keep-alive
cache-control: no-cache
The Content-Type header may be one of the issues, but I'm not certain how to exclude these "temporary headers" in postman.
I am generating the pre-signed URL in a lambda as follows:
public string FunctionHandler(Input input, ILambdaContext context)
{
_logger = context.Logger;
_key = input.key;
_bucketname = input.bucketname;
string signedURL = _s3Client.GetPreSignedURL(new GetPreSignedUrlRequest()
{
Verb = HttpVerb.PUT ,
Protocol = Protocol.HTTPS,
BucketName = _bucketname,
Key = _key,
Expires = DateTime.Now.AddMinutes(5)
});
returnObj returnVal = new returnObj() { url = signedURL };
return JsonConvert.SerializeObject(returnVal);
}
Your pre-signed url should be like https://bucket-name.s3.region.amazonaws.com/folder/filename.jpg?AWSAccessKeyId=XXX&Content-Type=image%2Fjpeg&Expires=XXX&Signature=XXX
You can upload to S3 with postman by
Set above url as endpoint
Select PUT request,
Body -> binary -> Select file
I was able to get this working in Postman using a POST request. Here are the details of what worked for me. When I call my lambda to get a presigned URL here is the json that comes back (after I masked sensitive and app-specific information):
{
"attachmentName": "MySecondAttachment.docx",
"url": "https://my-s3-bucket.s3.amazonaws.com/",
"fields": {
"acl": "public-read",
"Content-Type": "multipart/form-data",
"key": "attachment-upload/R271645/65397746_MySecondAttachment.docx",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "WWWWWWWW/20200318/us-east-1/s3/aws4_request",
"x-amz-date": "20200318T133309Z",
"x-amz-security-token": "XXXXXXXX",
"policy": "YYYYYYYY",
"x-amz-signature": "ZZZZZZZZ"
}
}
In Postman, create a POST request, and use “form-data” to enter in all the fields you got back, with exactly the same field names you got back in the signedURL shown above. Do not set the content type, however. Then add one more key named “file”:
To the right of the word file if you click the drop-down you can browse to your file and attach it:
In case it helps, I’m using a lambda written in python to generate a presigned URL so a user can upload an attachment. The code looks like this:
signedURL = self.s3.generate_presigned_post(
Bucket= "my-s3-bucket",
Key=putkey,
Fields = {"acl": "public-read", "Content-Type": "multipart/form-data"},
ExpiresIn = 15,
Conditions = [
{"acl": "public-read"},
["content-length-range", 1, 5120000]
]
)
Hope this helps.
Your pre-signed url should be like https://bucket-name.s3.region.amazonaws.com/folder/filename.jpg?AWSAccessKeyId=XXX&Content-Type=image%2Fjpeg&Expires=XXX&Signature=XXX
You can upload to S3 with postman by
Set above url as endpoint
Select PUT request,
Body -> binary -> Select file
I was facing the same problem and below is how it worked for me.
Note, I am making signed URL by using AWS S3 Java SDK as my backend is in Java. I gave content type as "application/octet-stream" while creating this signed Url so that any type of content can be uploaded. Below is my java code generating signed url.
public String createS3SignedURLUpload(String bucketName, String objectKey) {
try {
PutObjectRequest objectRequest = PutObjectRequest.builder().bucket(bucketName).key(objectKey)
.contentType("**application/octet-stream**").build();
S3Presigner presigner = S3Presigner.builder().region(s3bucketRegions.get(bucketName))
.credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials)).build();
PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(presignedURLTimeoutInMins)).putObjectRequest(objectRequest)
.build();
PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest);
return presignedRequest.url().toString();
} catch (Exception e) {
throw new CustomRuntimeException(e.getMessage());
}
}
## Now to upload file using Postman
Set the generated url as endpoint
Select PUT request,
Body -> binary -> Select file
Set header Content-Type as application/octet-stream (This point I was missing earlier)
It's actually depends in how you generated URL,
If you generated using JAVA,
Set the generated url as endpoint
Select PUT request,
Body -> binary -> Select file
If you generated using PYTHON,
Create a POST request, and use form-data to enter in all the fields you got back, with exactly the same field names you got back in the signedURL shown above.
Do not set the content type, however. Then add one more key named “file”:
Refer the accepted answer picture

Google Cloud Storage: CORS settings doesn't work for signed URLs

The response of PUT request with signed URL doesn't contain header Access-Control-Allow-Origin.
import os
from datetime import timedelta
import requests
from google.cloud import storage
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = <path to google credentials>
client = storage.Client()
bucket = client.get_bucket('my_bucket')
policies = [
{
'origin': ['*'],
'method': ['PUT'],
}
]
bucket.cors = policies
bucket.update()
blob = bucket.blob('new_file')
url = blob.generate_signed_url(timedelta(days=30), method='PUT')
response = requests.put(url, data='some data')
for header in response.headers.keys():
print(header)
Output:
X-GUploader-UploadID
ETag
x-goog-generation
x-goog-metageneration
x-goog-hash
x-goog-stored-content-length
x-goog-stored-content-encoding
Vary
Content-Length
Date
Server
Content-Type
Alt-Svc
As you can see there is no CORS-headers. So, can I conclude that GCS doesn't support CORS properly/fully?
Cross Origin Resource Sharing (CORS) allows interactions between resources from different origins. By default, in Google Cloud Storage it is prohibited/disabled in order to prevent malicious behavior.
You can enable it either using Cloud Libraries, Rest API or Cloud SDK, keeping in mind following rules:
Authenticate using user/service account with the permissions for Cloud Storage type: FULL_CONTROL.
Using XML API to get proper CORS headers, use one of the two URLs:
- storage.googleapis.com/[BUCKET_NAME]
- [BUCKET_NAME].storage.googleapis.com
Origin storage.cloud.google.com/[BUCKET_NAME] will not respond with CORS header.
Request need proper ORIGIN header to match bucket policy ORIGIN configuration as stated in the point 3 of the CORS troubleshooting documentation, in case of your code:
headers = {
'ORIGIN': '*'
}
response = requests.put(url, data='some data', headers=headers)
for header in response.headers.keys():
print(header)
gives following output:
X-GUploader-UploadID
ETag
x-goog-generation
x-goog-metageneration
x-goog-hash
x-goog-stored-content-length
x-goog-stored-content-encoding
Access-Control-Allow-Origin
Access-Control-Expose-Headers
Content-Length
Date
Server
Content-Type
I had this issue. For me the problem was I was using POST instead of PUT. Furthermore, I had to set the Content-Type of the upload to match the content type used to generate the form. The default Content-Type in the demo is "application/octet-stream", so I had to change it to be whatever was the content type of the upload. When doing the XMLHttpRequest, I just had to send the file directly instead of using FormData.
This was how I got the signed url.
const options = {
version: 'v4',
action: 'write',
expires: Date.now() + 15 * 60 * 1000, // 15 minutes
contentType: 'application/octet-stream',
};
// Get a v4 signed URL for uploading file
const [url] = await storage
.bucket("lsa-storage")
.file(upload.id)
.getSignedUrl(options as any);

Retuning stream in AWS API Gateway -> Lambda function?

I have created an API using AWS api gateway like https://api.mydomain.com/v1/download?id=1234". The download resource has GET method. And the GET method is invoking lambda function using Lambda Proxy Integration.
The Lambda function needs to act as Proxy. It needs to resolve correct backend endpoint based on header x-clientId and then forward the request to that backend endpoint and return response as it is. So it needs to be generic to handle GET request of different content-type.
My lambda function looks like ( .NET Core)
public async Task<APIGatewayProxyResponse> Route(APIGatewayProxyRequest input, ILambdaContext context)
{
var clientId = headers["x-clientId"];
var mappings = new Mappings();
var url = await mappings.GetBackendUrl(clientId, input.Resource);
var httpClient = new HttpClient();
var response = await httpClient.GetAsync(url);
response.EnsureSuccessStatusCode();
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = false,
Body = await response.Content.ReadAsString())
};
}
The handler above works as long as request and response's content-type is application/json or application/xml. However i am not sure how to handle response when backend returns stream.
For download API, the backend returns Content-Disposition: attachment; filename="somefilename and ContentType may be one of the following:
application/pdf
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/x-zip-compressed
application/octet-stream
For these streams, How do i set APIGatewayProxyResponse.Body?
For Excel file I have tried setting body like below
var proxyResponse = new APIGatewayProxyResponse()
{
Headers = new Dictionary<string, string>(),
StatusCode = (int)System.Net.HttpStatusCode.OK,
IsBase64Encoded = true,
Body = Convert.ToBase64String(await response.Content.ReadAsByteArrayAsync())
};
proxyResponse.Headers.Add("Content-Type", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
proxyResponse.Headers.Add("Content-Disposition", "attachment; filename=\"Report.xlsx\"");
When i access the Url from the browser and try to open the file. I get error
Excel cannot open the fileReport.xlsxbecuase the file format or file extension is not valid. Verify that the file has not been corrupted and that the extention matches the format of the file
I think the issue is how i am setting the response body
Update 1
So based on AWS doc Binary Data Now Supported by API Gateway. Now as per the documentation
you can specify if you would like API Gateway to either pass the
Integration Request and Response bodies through, convert them to text
(Base64 encoding), or convert them to binary (Base64 decoding). These
options are available for HTTP, AWS Service, and HTTP Proxy
integrations. In the case of Lambda Function and Lambda Function Proxy
Integrations, which currently only support JSON, the request body is
always converted to JSON.
I am using Lambda Function Proxy, which currently support JSON. However the example here shows how to do it with Lambda Proxy.
I think what i am missing here is Binary Media Types setting and Method Response settings. Below is my setting. Not sure if these settings are correct
Binary Media
Method Response
here how solved it
1>add Binary Media Types. API->Settings->Binary Media Types -> add
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
2>In Method Response Add Content-Disposition and Content-Type headers for thestatus 200
3>In Integration Response map these headers to headers that are coming from the backend. And also set content handling convert to binary. (our backend api is returning file blob in body)

Invalid Date When Uploading to AWS S3

I'm trying to upload an image to S3 via putObject and a pre-signed URL.
Here is the URL that was provided when I generated the pre-signed URL:
https://<myS3Bucket>.s3.amazonaws.com/1ffd1c88-5661-48f9-a135-04bd569614dd.jpg?AWSAccessKeyId=<accessKey>&Expires=1458177431311&Signature=<signature>-amz-security-token=<token>
When I attempt to upload the file via a PUT, AWS responds with:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Invalid date (should be seconds since epoch): 1458177431311</Message>
<RequestId>...</RequestId>
<HostId>...</HostId>
</Error>
Here is the curl version of the request I was using:
curl -X PUT -H "Cache-Control: no-cache" -H "Postman-Token: 78e46be3-8ecc- 4156-be3d-7e2f4688a127" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "file=#[object Object]" "https://<myS3Bucket>.s3.amazonaws.com/1ffd1c88-5661-48f9-a135-04bd569614dd.jpg?AWSAccessKeyId=<accessKey>&Expires=1458177431311&Signature=<signature>-amz-security-token=<mySecurityToken>"
Since the timestamp is generated by AWS, it should be correct. I have tried changing it to include decimals and got the same error.
Could the problem be in the way I'm uploading the file in my request?
Update - Add code for generating the signed URL
The signed URL is being generated via the AWS Javascript SDK:
var AWS = require('aws-sdk')
var uuid = require('node-uuid')
var Promise = require('bluebird')
var s3 = new AWS.S3()
var params = {
Bucket: bucket, // bucket is stored as .env variable
Key: uuid.v4() + '.jpg' // file is always a jpg
}
return new Promise(function (resolve, reject) {
s3.getSignedUrl('putObject', params, function (err, url) {
if (err) {
reject(new Error(err))
}
var payload = { url: url }
resolve(payload)
})
})
My access key and secret key are loaded via environment variables as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Amazon S3 only supports 32 bit timestamps in signed urls:
2147483647 is the largest timestamp you can have.
I was creating timestamps 20 years in the future and it was breaking my signed URL's. Using a value less than 2,147,483,647 fixes the issue.
Hope this helps someone!
You just get bitten by bad documentation. Seems relate to this link
Amazon S3 invalid date when using expires in url_for
The integer are mean for " (to specify the number of seconds after the current time)".
Since you enter the time in epoch(0 = 1970-1-1), So it is current epoch time + your time, 46+46 years= 92 years. Appear to be crazy expiration period for s3.