I am using AWS API Gateway put method to post an image to S3 bucket. In API Gateway settings when I add / in Binary Media Types I get the following error message
Access to fetch at ‘some_invokation_request/my_s3_bucket/image.png’ from origin ‘http://localhost:3001’ has been blocked by CORS policy: Response to preflight request doesn’t pass access control check: It does not have HTTP ok status.
When I remove / from Binary media types in settings I can post an image and when I check the posted image in S3 it's messed up.
I'll post the original image and S3 image
Apparently this is an open issue right now and you can find further details here
I believe you have two different errors present. One is CORS related and the other is the upload of an image getting messed up. I don't think they are necessarily related. I've hit the same issue with the image upload showing the tiny white square but faced no CORS issue.
For me, the problem turned out to be trying to combine the use of Convert to Binary (if needed) and a mapping template on the Integration Request. According to the documentation (and my experience as well), if a mapping template is defined, APIGW doesn't run the Convert to Binary passthrough functionality. What gets passed to the S3 bucket for saving is a base64 encoded string instead of actual binary, thus the messed up image.
Excerpt from the linked documentation:
"On the contentHandling property of the Integration resource, set CONVERT_TO_BINARY. Set WHEN_NO_MATCH as the passthroughBehavior property value without defining a mapping template. This enables API Gateway to invoke the passthrough template."
Related
I have created a React Application to upload a file to the S3 bucket. To do this, I have created a PUT API from the AWS API Gateway and the API is working well in the POSTMAN. The problem is, when I call the API from my React Application locally (http://localhost:3000), I'm getting a 500 error from the OPTIONS request (Preflight).
I have set the Binary Media Types to "image/jpeg" and When I remove the Binary Media Types, the API returns the 200 response, but then, an empty image is uploaded without its content.
How can I solve this?
This is my Request Integrations of both the PUT method and the OPTIONS method
PUT
OPTIONS
Binary Media Types
We have enabled AWS WAF solution before my ALB and have SQL injection and XSS detection enabled. We have tried to setup a custom rule to check if the content-type is multipart/form-data* using regex.
We have set that custom rule with higher priority. When using the custom rule the images are uploaded but the script tags are not forbidden. Without having the custom rule if we try uploading the images one particular image alone is not getting uploaded and throws 403 forbidden.
Any hints on adding XSS and custom rule to allow image uploads?
Goto your Web ACL and click on edit AWS-AWSManagedRulesCommonRuleSet and make Override rules action to True for rule SizeRestrictions_BODY
Check your image metadata. I recently encountered this issue, and was getting the "GenericRFI_BODY" error in the ACL logs. It turns out the test image I was uploading had an illegal path in its exif data. There was a URL that pointed to the site where the image came from in some metadata field, and the "://" pattern in that URL was triggering the rule. Stripping the metadata from the image allowed it to upload.
I strongly discourage base64 encoding to circumvent firewall rules. This will bloat the size of your files, and multipart/form-data exists specifically to stream large binaries back and forth from client to server - not to post massive serialized text blocks.
Here's the RFC: https://www.ietf.org/rfc/rfc2388.txt
I faced 403 issue in AWS firewall when I try to add image as multipart/form-data.
Some of the WAF rules which blocks the image upload are, AWS#AWSManagedRulesSQLiRuleSet#GenericRFI_BODY, AWS#AWSManagedRulesSQLiRuleSet#SQLi_BODY and AWS#AWSManagedRulesCommonRuleSet#CrossSiteScripting_BODY.
I solved this issue by uploading the image as base64 string instead of uploading as multipart/form-data.
In my case this was due to the following WAF rule:
ruleGroupList.0.ruleGroupId AWS#AWSManagedRulesCommonRuleSet
ruleGroupList.0.terminatingRule.ruleId SizeRestrictions_BODY
And I solved the problem by overriding the default rule action from BLOCK to CHALLENGE
Problem Statement:- Not able to stream PDF, using AWS gateway/Lambda setup.
I've following Setup:-
Gateway API -> Lambda Function(java) -> S3
Api should stream a PDF back to the client via Lambda from S3 server.
I am unable to do so.
Things tried
1) On Lambda side
Use RequestStreamHandler, write PDF on OutputStream
outputStream.write();
Tried Setting Base64 encoding using following
AWS Base64 Utility
java.util Base64 Utility
apache Base64 utility
Send Byte[] without encoding
Use non-proxy handler, write PDF as string to one of the output variable.
Tried Setting Base64 encoding using following
AWS Base64 Utility
java.util Base64 Utility
apache Base64 utility
Send Byte[] without encoding
2) On Gateway Side
Set Proxy integration
Remove Proxy integration
Tried Set Content Handling using
Passthrough
Convert to Binary
Convert to String
Set headers
Content-Type = 'application/pdf'
Content-Disposition = 'attachment; filename="nameofpdffile.pdf"'
I've tried all the permutations/combinations of these 2. However, I am not able to get it to work and documentation around this area seems poor.
When I set Base64 encoding in Lambda and on gateway side specify content handling as convert to binary, i get error, fail to conert using Base64 decode.
For other combinations, I get binary data as output when I test it using gateway Test funciton, however my client(Postman, Chrome, Safari) fail to convert it to PDF
Note:- I've a workaround in place, where I create a signedURL to S3 object, and send it as redirect. However, I am trying to make it work by streaming a binary file using Lambda.
Thanks for reading such a long post. Any pointers, links in the right direction are highly appreciated.
- Frustated AWS user :-)
I had a similar problem with trying to offer up gzipped content: Client -> API-Gateway -> S3.
I couldn't figure out why the content I pulled couldn't get recognized by the client (either browser or code) until I inspected the header(the binary header, not http).
It appears that API Gateway by default assumes a string based encoding (utf-8) so what I was seeing is data that API Gateway was transforming on the fly and adding utf-8 headers in various places.
Late last year API Gateway started to support binary payloads. I started reading this article which may help in your case. It did not in mine, but your use case is slightly different and closer to the article so it might.
https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/
As for what I ended up doing... I realized I was just using the API Gateway as a passthrough (to take advantage of the Cognito based authentication I had already setup) so I bypassed API Gateway all together and used the aws js sdk to connect directly to s3 (and set IAM policies on the bucket separately)
So I have CORS enabled going through the basic setup given by AWS Gateway. However for this API I need to allow Control Origins for all requests and allow credentials.
Here is what it looks like
The issue as you may have guessed is this setup is not allowed by CORS, you can not have a wildcard for Origin and have credentials as true. Normally the work around for this is to just grab the requesting domain and add it into the Origin Header. Which is more or less what I want to do. But I don't know how to get that information and add it as a mapping value. Where does API Gateway store that information and how do i get it?
UPDATE:
I have to pass through HTTP Header Host to my Lambda Function which I should have mentioned earlier, I have tried implementing the Answer below but I cannot access the header to pass it to the Lambda function using the instructions provided. Any more assistance with this is greatly appreciated.
Okay After hours of research and finding bits of information across the internet I have a solution and hopefully it is useful for other people.
To pass an HTTP Header that is not a default value provided by AWS API Gateway and then access that data via a Lambda Function and return that data in the Response Header follow the steps below
In "Method Request" go to "HTTP Request Headers" and add your desired header to capture. ie. if we want to get the host value of the API url you can enter "Host" here. if you want to get the website host of the caller use "Origin"
In "Integration Request" go to mapping templates and create a new template if an "application/json" does not exist, if it does just update it.
This is the important part, pass the header value you set in step 1. To do that write something similar to the following in the Template Box.
{
"origin" : "$input.params().header.Origin",
"host" : "$input.params().header.Host"
}
You can also pass in any url parameters you have defined in the same JSON.
Access the data from Lambda, The integration request passed the information into the "Event" parameter if using Node as the Lambda Backend code. to retrieve the value of any header just use the following within your handler.
event.origin;
When sending back your response from Lambda to API Gateway, it is best to format the response in JSON. Something similar to this.
{
"origin" : event.origin,
"host" : event.host,
"nonHeaderOutput" : "Hello World"
}
In "Integration Response" go to "Header Mappings", if the header you need is not listed you may add it in "Method Response" and it will then appear here. For this example I used "Access-Control-Allow-Origin" and edited the Mapping Value to be integration.response.body.origin
now go to "Mapping Templates and select the content type you want to use, and then edit the template to access the non header responses by adding this to the Template Box
$input.path("$.nonHeaderOutput")
That is it now the header that was sent to the API can be used in your method Response.
Jurgen from API Gateway here.
Thanks for reporting this issue. There is currently no simple way to set it up via the "Enable CORS" feature in the console. However, we are looking into it to improve the user experience for this use case.
As a potential workaround, you could passthrough the Origin header from the client to your backend and parse / create the value for the Access-Control-Allow-Origin header there. Then you would map the Header in the Integration Response from 'integration.response.header.Access-Control-Allow-Origin' to Access-Control-Allow-Origin and return it to the client.
Hope this helps.
Best,
Jurgen
I'm trying to set a Content-Security-Policy header for an html file I'm serving via s3/cloudfront. I'm using the web-based AWS console. Whenever I try to add the header:
it doesn't seem to respect it. What can I do to make sure this header is served?
I'm having the same problem (using S3/CloudFront) and it appears there is currently no way to set this up easily.
S3 has a whitelist of the headers permitted, and Content-Security-Policy is not on it. Whilst it is true you can use the prefixed x-amz-meta-Content-Security-Policy, this is unhelpful as there is no browser support for it.
There are two options I can see.
1) you can serve the html content from a webserver on an EC2 instance and set that up as another CloudFront origin. Not really a great solution.
2) include the CSP as a meta tag within your html document:
<!doctype html>
<html>
<head>
<meta http-equiv="Content-Security-Policy" content="default-src http://*.foobar.com 'self'">
...
This option is not as widely supported by browsers, but it appears to work with both Webkit and Firefox, so the current Chrome, Firefox, Safari (and IOS 7 Safari) seem to support it.
I chose 2 as it was the simpler/cheaper/faster solution and I hope AWS will add the CSP header in the future.
S3/CloudFront takes any headers that the origin set and forward those to the client, but you can't set custom headers on you response directly.
You can use Lambda#Edge function that can inject security headers through CloudFront.
Here is how the process works: (reference aws blog)
Viewer navigates to website.
Before CloudFront serves content from the cache it will trigger any
Lambda function associated with the Viewer Request trigger for that
behavior.
CloudFront serves content from the cache if available, otherwise it
goes to step 4.
Only after CloudFront cache ‘Miss’, Origin Request trigger is fired
for that behavior.
S3 Origin returns content.
After content is returned from S3 but before being cached in
CloudFront, Origin Response trigger is fired.
After content is cached in CloudFront, Viewer Response trigger is
fired and is the final step before viewer receives content.
Viewer receives content.
Below is the blog from aws on how to do this step by step.
https://aws.amazon.com/blogs/networking-and-content-delivery/adding-http-security-headers-using-lambdaedge-and-amazon-cloudfront/
If you are testing through CloudFront, have you made sure you have invalidated the cached objects? Can you try to upload a completely new file and then try accessing it via CF and see if the header is still not there?
Update
Seems like custom metadata will not work as expected as per DOC. Any metadata other than the ones supported by S3 (the ones displayed in the dropdown) will have to be prefixed with x-amz-meta-