I have a requirement to upload a CSV file using Rest API, process this file in AWS Lambda, then provide a response in API back to the user. How can I achieve this?
I thought of uploading the file to s3 using API and then triggering a lambda function based on s3 events, but in this case, I won't be able to provide a response to the user.
I am adapting this article to your question.
Create a PUT method for your API for uploading CSV file
Choose Actions, and then choose Create Method.
From the dropdown list, choose PUT and then choose the check mark icon.
Under the Integration type category, choose AWS Service.
For AWS Region, choose us-east-1 or the AWS Region you see on the Bucket properties page.
For AWS Service, choose Simple Storage Service (S3).
Leave AWS Subdomain empty.
For the HTTP method, choose PUT.
For Action Type, choose Use path override.
For Path override (optional), enter {bucket}/{key}.
For the Execution role, enter the Amazon Resource Name (ARN) for the IAM policy for the API Gateway role.
For Content Handling, choose Passthrough.
Choose Save.
Configure parameter mappings for the PUT method
In the Resources panel of your API page, choose PUT.
Choose Integration Request.
Expand URL Path Parameters.
Choose Add path.
For Name, enter bucket.
For Mapped from, enter Method.request.path.folder.
Choose the checkmark icon at the end of the row.
Repeat steps 4 through 7. In step 5, set Name to key. Step 6 sets Mapped from to Method.request.path.object.
Create a GET method for your API for retrieving a CSV file
In the Resources panel of your API page, choose /{object}.
Choose Actions, and then choose Create Method.
From the dropdown list, choose GET and then choose the check mark icon.
Under the Integration type category, choose AWS Service.
For AWS Region, choose us-east-1, or the Region you see on the Bucket properties page.
For AWS Service, choose Simple Storage Service (S3).
Leave AWS Subdomain empty.
For the HTTP method, choose GET.
For Action Type, choose Use path override.
For Path override (optional), enter {bucket}/{key}.
For the Execution role, enter the Amazon Resource Name (ARN) for the IAM policy for the API Gateway role.
For Content Handling, choose Passthrough.
Choose Save.
Configure parameter mappings for the GET method
In the Resources panel of your API page, choose GET.
Choose Integration Request.
Expand URL Path Parameters.
Choose Add path.
For Name, enter bucket.
For Mapped from, enter Method.request.path.folder.
Choose the checkmark icon at the end of the row.
Repeat steps 4 through 7. In step 5, set Name to key. Step 6 sets Mapped from to Method.request.path.object.
Set up response mapping to see the CSV in the browser
In the Resources panel of your API page, choose GET.
Choose Method Response.
Expand 200.
Under Response Body for 200, remove application/JSON.
Under Response headers for 200, choose Add header.
For Name, enter content type.
Choose the check mark icon to save.
Choose Method execution to go back to the Method Execution pane.
Choose Integration Response.
Expand 200, and then expand Header Mappings.
Choose the pencil icon at the end of the row named content-type.
Enter text/CSV to see a CSV file.
Set up binary media types for the API
In the navigation pane of your API page, choose Settings.
In the Binary Media Types section, choose Add Binary Media Type.
In the text box, add the following string: /
Note: Avoid putting the string in quotes. You can substitute a wildcard for a particular Multipurpose Internet Mail Extensions (MIME) type you want to treat as a binary media type. For example, choose "image/jpeg" to have API Gateway treat JPEG images as binary media types. If you add /, then API Gateway treats all media types as binary media types.
Choose Save Changes.
Resolve CORS error with binary settings for API
If you want to use the aforementioned APIs (PUT and GET) in a web application, you might encounter a CORS error.
To resolve the CORS error with binary settings turned on, start CORS from the API Gateway console.
In the Resources panel of your API page, select /{object}.
For Actions, choose Enable CORS.
Choose Enable CORS and replace existing CORS headers.
Use the update integration CLI command to update the content handling property. This update allows the binary content to be handled for preflight options requests with mock integration.
Run the two CLI commands that follow by updating the API ID, Resource ID, and AWS Region. You can obtain the API ID and Resource ID from the top of the Gateway API console when you select the {object} resource.
Finally, deploy your API and upload a CSV file via your API
curl -i --location --request PUT 'https://abc12345.execute-api.us-west-2.amazonaws.com/v1/testfolder/test.csv' --header 'Content-Type: text/csv' --data-binary '#/Path/to/file/test.csv'
Related
I am using sitespeed.io to measure performance of some of my static URL's and uploading the results into AWS S3 bucket.
Inside AWS S3 bucket it is structured exactly how sitespeed outputs the data. I have attached an image for it and when you go inside that folder what you see is index.html which shows all the details of the pages configured.
Now I have a page which has a calendar and when you select any date, it should show all the folders of that date. When you click on that folder it should open the sitespeed.io summary page. I've attached that as well. Unfortunately I can't share anything else from the code.
What i want to know is it possible to trigger a lambda function using a rest API which will fetch the result from S3 or dynamoDB and display on the front end for user to view?
I've deployed an Angular static website to Amazon S3 and used Cloudfront for it. When I'am accessing the default page myurl.com/ everything works fine and I can navigate through pages. But if I go to a certain page directly myurl.com/posts for example I get an access denied with cloudfront. If I do it with the amazon s3 website endpoint I get a 404 response.
S3
Cloudfront
The documentation states that a 403 will be returned in CloudFront for a number of reasons. One of the resolutions is The requested objects must exist in the bucket..
As you're trying to access an object that does not exist CloudFront may be turning this into the 403 that you are receiving.
You should attempt accessing a key that does exist only, and configure your s3 endpoint to use an error document for times when the page does not exist.
in my case, I set a default root object and things worked, after waiting for about 10 minutes so the setting changes sync.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/DefaultRootObject.html
To specify a default root object using the CloudFront console:
Sign in to the AWS Management Console and open the Amazon CloudFront console at https://console.aws.amazon.com/cloudfront/.
In the list of distributions in the top pane, select the distribution to update.
In the Distribution Details pane, on the General tab, click Edit.
In the Edit Distribution dialog box, in the Default Root Object field, enter the file name of the default root object.
Enter only the object name, for example, index.html. Do not add a / before the object name.
To save your changes, click Yes, Edit.
I have example-bulkorder.com domain which is connected to ecommerce hosting (Business Catalyst).
This site doesn't need ecommerce anymore so I create a simple page on example.com/bulkorder which has a form that send email to me so that I can handle bulk orders manually via email.
At the moment I just set up redirection with javascript but when I unsubscribe Business Catalyst hosting, I will need to find a way to set up this redirection from example-bulkorder.com domain to example.com/bulkorder
Is there a way to do this with AWS or should I just kill the old domain?
I am not worried about all pages under the domain but at least the main url to be redirected.
Or should I just let the old domain goes to broken link and let people to find out that the url is not available? What's the best solution? I want to keep the old domain in case I need it with separate ecommerce system in the future.
Use the following procedure to redirect your domain
Requirements
A hosted zone for the domain example-bulkorder.com is hosted in Amazon Route 53.
You have permissions to add resource records to the hosted zone of example.com.
You have permissions to create an Amazon S3 bucket.
You are able to create an S3 bucket with the exact name for example.com.
Note: The sites must be HTTP because the redirect cannot connect to S3 over HTTPS.
In the Amazon S3 console, create an S3 bucket with the exact name as example-bulkorder.com.
Note: S3 bucket names must be globally unique. If the bucket name you need is already in use, this solution cannot be used.
Select the bucket you just created, and then choose Properties. Expand the Static Website Hosting drop-down menu, and then choose Redirect all requests to another hostname.
In the Redirect, all requests to another hostname field, enter example.com/bulkorder and then choose Save.
In the Route 53 console, select the hosted zone named example-bulkorder.com.
Create a resource record for example.com with the following values:
Record Type: Choose A IPv4 address.
Alias: Choose Yes.
Alias Target: Select the blank value field, expand the drop-down menu, and then choose your S3 bucket under the heading S3 Website Endpoints.
Routing Policy: Choose Simple.
Evaluate Health Target: Choose No and then choose to Create.
Note: The drop-down menu might take a few moments to populate your bucket name.
You should now be able to redirect example-bulkorder.com to example.com/bulkorder.
You could do this serverless by using Cloudflare (probably the easiest way), just need to create a page rule to redirect your domain to your desired endpoint.
Check this post: How do I perform URL forwarding or redirects with Cloudflare?
Within AWS is also possible you have more options, one of this is using an S3 bucket, check this article: https://aws.amazon.com/premiumsupport/knowledge-center/redirect-domain-route-53/
Can I allow a 3rd party file upload to an S3 bucket without using IAM? I would like to avoid the hassle of sending them credentials for an AWS account, but still take advantage of the S3 UI. I have only found solutions for one or the other.
The pre-signed url option sounded great but appears to only work with their SDKs and I'm not about to tell my client to install python on their computer to upload a file.
The browser based upload requires me to make my own front end html form and run in on a server just to upload (lol).
Can I not simply create a pre-signed url which navigates the user to the S3 console and allows them to upload before expiration time? Of course, making the bucket public is not an option either. Why is this so complicated!
Management Console
The Amazon S3 management console will only display S3 buckets that are associated with the AWS account of the user. Also, it is not possible to limit the buckets displayed (it will display all buckets in the account, even if the user cannot access them).
Thus, you certainly don't want to give them access to your AWS management console.
Pre-Signed URL
Your user does not require the AWS SDK to use a pre-signed URL. Rather, you must run your own system that generates the pre-signed URL and makes it available to the user (eg through a web page or API call).
Web page
You can host a static upload page on Amazon S3, but it will not be able to authenticate the user. Since you only wish to provide access to specific people, you'll need some code running on the back-end to authenticate them.
Generate...
You ask: "Can I not simply create a pre-signed url which navigates the user to the S3 console and allows them to upload before expiration time?"
Yes and no. Yes, you can generate a pre-signed URL. However, it cannot be used with the S3 console (see above).
Why is this so complicated?
Because security is important.
So, what to do?
A few options:
Make a bucket publicly writable, but not publicly readable. Tell your customer how to upload. The downside is that anyone could upload to the bucket (if they know about it), so it is only security by obscurity. But, it might be a simple solution for you.
Generate a very long-lived pre-signed URL. You can create a URL that works for months or years. Provide this to them, and they can upload (eg via a static HTML page that you give them).
Generate some IAM User credentials for them, then have them use a utility like the AWS Command-Line Interface (CLI) or Cloudberry. Give them just enough credentials for upload access. This assumes you only have a few customers that need access.
Bottom line: Security is important. Yet, you wish to "avoid the hassle of sending them credentials", nor do you wish to run a system to perform the authentication checks. You can't have security without doing some work, and the cost of poor security will be much more than the cost of implementing good security.
you could deploy a lambda function to call "signed URL" then use that URL to upload the file. here is an example
https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/
We are using the s3 server to allow users to download large zip files a limited number of times. We are searching for a better method of counting downloads that just counting button clicks.
Is there anyways we can give our user a signed url to temporary download the file (like we are doing now) and check that token with amazon to make sure the file was successfully downloaded?
Please let me know what you think
You could use Amazon S3 Server Access Logging:
In order to track requests for access to your bucket, you can enable access logging. Each access log record provides details about a single access request, such as the requester, bucket name, request time, request action, response status, and error code, if any.
There is no automatic ability to limit the number of downloads via an Amazon S3 pre-signed URL.
A pre-signed URL limits access based upon time, but cannot limit based upon quantity.
The closest option would be to provide a very small time window for the pre-signed URL, with the assumption that only one download would happen within that time window.