I am trying to perform a document upload using an S3 pre-signed PUT url. I generated the url using java AWS SDK (GeneratePresignedUrlRequest.java). This url generation code sits in a lambda function behind AWS API gateway.
However I am getting the following error when I copy the generated url in Postman & try to perform an upload.
<Error>
<Code>AccessDenied</Code>
<Message>
There were headers present in the request which were not signed
</Message>
<HeadersNotSigned>host</HeadersNotSigned>
<RequestId>6E624EC66091A099</RequestId>
<HostId>
9HnUUZpa9zUWfyzuNrZtRkoN2CYxH8V6rqx5QgsMFC6foTZKuG5qmZL5ThnUXooYPk2reFzKKUg=
</HostId>
</Error>
The generated url is "https://<bucket>.s3.amazonaws.com/<key>?X-Amz-Security-Token=<value>&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20180718T013339Z&X-Amz-SignedHeaders=content-type%253Bhost&X-Amz-Expires=1799&X-Amz-Credential=<value>&X-Amz-Signature=<value>
Any suggestions as to what needs to be corrected here in the generation of url ?
The error was indeed related to a 'missing header', but not 'host' which was misleading. The value 'content-type' that comes before 'host' in X-Amz-SignedHeaders=content-type%253Bhost , needs to be explicitly provided as a request header when a http client uses the pre-signed url to upload a file to S3.
I had generated the pre-signed url in java as
GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName,objectKey).withMethod(HttpMethod.PUT).withContentType("application/pdf").withExpiration(expiration);
So S3 expects content-type with value 'application/pdf' to be part of the pre-signed url request as a header for a signature match as I understand.
Related
Using Adobe 'PDF services API'
Generated AssetID and presigned URI using https://pdf-services-ue1.adobe.io/assets API.
Got following as response.
{
"uploadUri": "https://dcplatformstorageservice-prod-eu-west-1.s3-accelerate.amazonaws.com/16a9de6d208743239d1f694295f12626_254127B763EDC88B0A495FC7%40techacct.adobe.com/9fea6bbd-4f94-40d2-8463-d4e94aaa8424?X-Amz-Security-Token=FwoGZXIvYXdzEAIaDIBucznIV%2BQEMQw0PCLTAczK0ByJ4MmpHzLcmN3VNzE%2Fuhh2sH3Xw7009oBaypEOWod2yD%2FtfwLKFwS8Lq0sSmXyJFpd4vU67PaHAUdtxuJt4vXIaL9t%2FNFtTy66TZz4lTD4rRQQP%2BgJNO6C7f3JOlVeIyVTSyGieQ3jXSwkctCHmAA7YeWqKutr83tCUPeVzrPRyqNOx7BMA6vQUSpTelIVvY9F67s0IOYtuETwZeE0XFh%2BfKA%2FeN3rEdRGqqq60aUVNQTZZCnqInpFHQ8xu%2FcQM6fFPnUO79IVUC7fqpV4xMson7bHnwYyLThcwrmnyKJyCNvmZwl%2BNUbkpVMmY0lCcMMTtXYRtxxTV8yYq7zzcTWicI6XBA%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230219T082707Z&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Expires=3600&X-Amz-Credential=ASIAWD2N7EVPJNDYTYLB%2F20230219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=c291f06087e85b5221e2910aba53bf4ce40e12542e800c7f159f009d3bfa94c6",
"assetID": "urn:aaid:AS:EW1:2ceb2ab0-e94b-4664-9bfa-0789e3fe111d"
}
Then uploaded PDF file to URI by making a PUT request with URI as the endpoint and uploading the file in binary in body. The response status of this request is 200 OK.
Then used the assetID for making a POST request 'https://pdf-services-ue1.adobe.io/operation/exportpdf'.
Can someone explain why 'asset not found', when PDF file successfully uploaded to presigned URI.
I tried to convert PDF to docx using adobe pdf services API, by making API calls using postman.
But got error 404 'Assset not found', despite pdf file being successfully uploaded to the presigned URI.
I have a process where I get a signed url for a bucket that expires in 5 minutes. As soon as I get the signed url I start to upload the file to the bucket.
Lets say, it takes 6 minutes to fully upload the file. What will happen?
Is the expiration time referring to when the action (e.g. download, upload) can begin or is it for the action to complete?
Is the expiration time referring to when the action (e.g. download, upload) can begin or is it for the action to complete?
Both :-)
When the action is atomic and it was started when the link was valid, this action will be completed even if the link became not valid during the process.
However if the action is mutli-step (e.g. multi part file upload), then the step that is performed with invalid link will be failed.
Source of truth.
Based on you use case, I simulated the environment in such a way that while initiating the upload, there was no error but as soon as the url expires upload fails.
So below is the response I get when I upload using a presigned URL and while uploading the url getting expired:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Request has expired</Message>
<Expires>2019-12-13T06:45:29Z</Expires>
<ServerTime>2019-12-13T06:45:33Z</ServerTime>
<RequestId>8D0F75A23E9A1EC2</RequestId>
<HostId>CNhlDK4RRVowLZSHdSF7pPJksIhnZdh99M3Sc9aBqnyCxLDlGG3XYMV0GMIdpz1pgM+HKLxsKVE=</HostId>
</Error>
So it is clear that, presigned url irrespective of POST or PUT, while uploading if the url is getting expired there, will be no file getting created in AWS s3 bucket console, and also AWS will give a response telling Request Expired
Currently my signed URLs all look like below and they work fine:
https://cloudfront/filekey?response-content-disposition=inline%3Bfilename%3D{utf-8-encoded-filename.pdf}&Expires=...
But some file names include unicode characters, so CloudFront would return:
<Error>
<Code>InvalidArgument</Code>
<Message>
Header value cannot be represented using ISO-8859-1.
</Message>
<ArgumentName>response-content-disposition</ArgumentName>
<ArgumentValue>inline;filename=文件名 - abc.pdf</ArgumentValue>
<RequestId>1234567890</RequestId>
<HostId>
123456789
</HostId>
</Error>
So as many people suggested (e.g. AWS Developer Forums), I added *=uft-8'' in response-content-disposition as below:
https://cloudfront/filekey?response-content-disposition=inline%3Bfilename%2A%3DUTF-8%27%27{utf-8-encoded-filename.pdf}&Expires=...
Now it returns
<Error>
<Code>AccessDenied</Code>
<Message>Access denied</Message>
</Error>
Its origin domain name is in this format: AWSDOC-EXAMPLE-BUCKET.s3.eu-west-1.amazonaws.com, so it's a S3 REST API endpoint.
origin access identity (OAI) is configured with existing identity. And I chose to manually update permissions on the S3 bucket.
In S3, objects are AES-256-encrypted. In bucket policy, there is only one statement to allow OAI access. The owner of this bucket and objects is the same account.
After reading I’m using an S3 REST API endpoint as the origin of my CloudFront distribution. Why am I getting 403 Access Denied errors?, I start to think maybe it's because the filename doesn't match?
The file uploaded was named 文件名 - abc.pdf, but in s3 it's shown as _-_abc.pdf. But I've tried to use _-_abc.pdf in response-content-disposition, it still returns me access denied.
I finally made it work, so I think I'd better write my solution here.
When an object is uploaded to S3, its filename will be encoded, as I mentioned in the question: 文件名 - abc.pdf -> _-_abc.pdf. But the filename saved in our database is still the original one, so it would return access denied as the filename is wrong.
But if I use _-_abc.pdf in response-content-disposition, it still returns access denied. Because I forgot I'm using signed URL, and it's signed based on object key and response-content-disposition.
So here I don't need adding utf-8 at all. And I should modify my url generation codes, instead of simply changing response-content-disposition in the browser.
Besides, because I'm not sure how S3 encode filenames, I directly extract filenames from object keys. Below is my codes written in Scala:
// r.s3ObjectKey is in format: YYYY/MM/DD/encodedFileName-digits
val fileName = r.s3ObjectKey.split("/").last.split("-").dropRight(1).mkString("-")
Is it possible to download Zip file present in S3 using API Gateway alone.
Configurations:
Integration type : AWS Service (S3)
endpoint : GET
Content-Type: application/zip (or) application/octet-stream
A corrupted zip file is getting downloaded.
I could able to do a workaround using S3 presigned url and don't want to make the bucket public.
The file can get corrupted when some headers are missing from the upload PUT request. Please make sure the file is not-corrupted by downloading it manually and checking. If it's corrupted from the GET request only please make sure you include all the needed headers in the GET request too.
I'm trying to authenticate a request using signature version 2. When I send the request to perform a GetObject action I get an error response asking for an expires header.
According to http://docs.aws.amazon.com/general/latest/gr/signature-version-2.html this should not be a problem.
Is aws trying to authenticate using sigv4 or is this perhaps a s3 GetObject specific problem?
In case there is interest in the response:
- <Error>
<Code>AccessDenied</Code>
- <Message>
Query-string authentication requires the Signature, Expires and AWSAccessKeyId parameters
</Message>
<RequestId>...</RequestId>
- <HostId>
...
</HostId>
</Error>