Im learning about AWS and Flutter to develop a mobile app (Android) like Instagram for a subject in the university.
My question is if there is a way to send an image from the app directly to a S3 Bucket and just store the URL related to that image in DynamoDB.
I’m using Flutter for the frontend (it doesn't use AWS SDK) and to develop the backend AWS Api Gateway (Api REST), Lambda and DynamoDB.
I read that the image can be passed in the request body if it is converted to base64 but I’m worried about if it is the optimal way to do it. I’m thinking that if I pass the image in that way the AWS free tier of Api Gateway and Lambda will be consumed because of the size of the request body that is passed and processed to be saved in a S3 Bucket considering that I will upload several images.
The dimensions of the image will be 1080x1350px so I think the size will be around 500KB and I'm planning to upload videos too.
I will be grateful for any kind of guidance because I am new to AWS.
You may use S3 presigned url as it will provide scalable solution which will overcome limits imposed by API Gateway. Because video file - particularly HD files, will introduce issues on limit w.r.t API Gateway later on if you pass file directly to API Gateway.
Related
I've got an jpeg image stored in a bucket using Amazon S3. I've followed this guide roughly to setup a REST API as an Amazon S3 Proxy in API Gateway. I'm using this API endpoint as way to access the images stored in the bucket using a GET request. I've also followed along with this guide that provides insight in enabling binary support. When I do a GET request at the given endpoint of the image, I want the image to show. The problem is that even following that guide, it doesn't work. The image doesn't show properly and I'm left with some junk:
Does anyone have any insight on this issue or a way they enabled binary support on their API endpoints?
I'm trying to create a reusable large-file serverless upload service in AWS (we host a number of sites). What I would like to do is to set up an API Gateway in AWS and use CORS to control which sites can upload, allowing the sites to use client-side code. Here is what I've tried and the roadblocks I've run into. Wondering if anybody has any suggested workarounds?
Calling S3 from client-code upload would require me to expose authentication information in client-side land, which seems bad
API Gateway does not appear to support calling S3 multipoint through its AWS Service integration type (URL is fixed to generic S3 service URL, and IAM isn't supported in HTTP integration type)
Leveraging Lambda to call the multipart API won't work, because it can only take in 6 MB of invoke request payload, and to get the 5 MB worth of minimal upload part size, base64 will make the data way more than 6 MB
I could do my own partial upload functionality in Lambda, storing the chunks in S3, but I can't figure out how to merge them together within Lambda's memory and tmp storage space (still PassThrough streams do not appear to work with AWS SDK)
Any ideas? Is any of these worth digging into? Or is serverless a no-go for this use case?
So, after further follow-up with Amazon, it's sort-of possible to use pre-signed URLs with the multipart API, but it's not very practical. Steps involved would include the following:
Create a new file, and split it into parts.
Generate a presigned URL to initiate the multiart upload.
Use the presigned URL to initiate the upload.
Generate a presigned URL for each part, using a part number.
Use the URLs to send the PutPart requests. Keep track of the Etag that is returned for the part number.
Combine all of the parts and corresponding ETAGs to form the request body.
Generate a presigned URL to complete the MP upload.
Complete the multipart upload by sending the request with the presigned complete multipart upload URL.
Will accept Angelo's answer since it did point in this direction which, technically, seems possible
You might be able to use presigned urls for the upload. In this case the client would hit your API, which would do whatever validation is necessary, and then generated a presigned url to S3 that is returned to the client. The client then directly uploads to s3.
You can see some information here: https://sanderknape.com/2017/08/using-pre-signed-urls-upload-file-private-s3-bucket/
Sorry for doing this kind of question.. but I´m a bit lost here....
I have an app which consist in an Angular4 as frontend and Java app as Backend.
But I´m planning to use AWS Lambda as I´m interested after seeing the videos in Amazon.
The issue is that I don´t know how to get the best from AWS.
My Java app has a very time consuming task to process some images (which takes several seconds).
But I'm not sure if I can deploy all my app in Lambda, or if the idea is to use a EC2 server and then the specific task for the image processing in the lambda. Can anyone please shed some light here?
Also, the frontend app can be deploy in a lambda, or again, lambda is just for specific task?
EDIT:
The application flow would be:
The user in the angular app upload an image, the image goes to the backend server in Java and it´s stored in (maybe) a AWS bucket.. Then the Java app with imagemagick process the image and the result is store in (maybe) another bucket.
So the question is when I need to use Lambda? just to convert the image or if the full backend (and maybe frontend) app would be there?<
I'm asking because I cannot find enough information about that...
First of all you can deploy your Angular frontend to Amazon S3. Also you can use AWS CloudFront to add custom domains and free SSL certificates from Amazon using Amazon Certificate Manager for your domain. For more details refer the article Deploying Angular/React Apps in AWS.
If you don't need to show tge image processing results immediately in frontend
For the image processing backend you can use AWS API Gateway and Lambda along with S3. For this recommended flow is you can use the API Backend to get an Signed URL or AWS STS in Lambda (Or Use Cognito Federated Identities) to get temporary access to Amazon S3 Bucket to Upload the image directly to S3 from Angular App. For more details on this refer the article Upload files Securely to AWS S3 Directly from Browser.
Note: AWS recently released a JavaScript Library called AWS Amplify to simplify the implementation of the above tasks.
After Uploading the image to S3 you can setup an event driven workflow by using Amazon S3 triggers to invoke an Lambda function to perform the image processing and save the process image back to S3 (If you need to store the result).
If you need to show the result immediately
Still use tge previous approach upto Upload to S3 from frontend and then invoke an API Gateway Lambda function passing the file path in S3 to process the image.
To understand the details in connecting both frontend and backend with AWS serverless technologies refer the article Full Stack Serverless Web Apps with AWS.
As a side note, you should be able to implement the required functionality with AWS Lambda without using AWS EC2.
I had plans to use the AWS API Gateway for three purposes. All of these endpoints are configured with custom domain names, with AWS issued SSL certificates and I have CNAME records configure to match the could front urls.
api.my-domain.com (REST api calls that return json data) (Working as expected)
images.my-domain.com (Proxy pass through of binary image data from S3) - Working as expected.
videos.my-domain.com (DOH!... )
Unfortunately dealing with videos I've run into a few issues. Smaller videos start to work but then generate an error. But.. that's not the main issue.
There is a 10MB max payload size on the response data from an API integration endpoint, so I must come up with another solution for the videos.
I don't want to host the images, or videos via cloundfront. And I want to use the same AWS issues wildcard certificate *.my-domain.com on all the endpoints. I wanted to use the API gateway for the image request because the images are small, and won't exceed the limit, and I can cache them at the api level.
a CNAME pointed to my video S3 bucket works, but can't use the same SSL certificate, and I wanted all traffic to originate vie the API gateway and not have request going directly to the bucket endpoint.
```
So.. what are my options?
It seems like my best option will be to transcode the MP4 videos to HLS, and host the S3 bucket via cloudfront. I hadn't really wanted to incur the charges of using cloudfont, but I don't see any better option, for the design I want.
The most recent videos will be viewed occasionally, not high demand, older videos will be viewed rarely so hosting them in cloudfront seems like a waste.
Typical setup for Video streaming in AWS is to Stream the Video stored in S3 through AWS CloudFront RTMP Distribution.
Going forward with CloudFront hosted content from my S3 bucket to see how it works.
The reason I picked CloudFront overall was the tight integration with other AWS services, and the ability to have complete control over the path/name of the assets in the S3 bucket, whereas with Vimeo you don't have very good control over the name of the assets.
I am trying to upload a file to s3 and then have lambda generate id, date.
I then want to return this data back to the client.
I want to avoid generating id and date on the client for security reasons.
Currently, I am trying to use API Gateway which invokes a lambda to upload into s3. However, I am having problems setting this up. I know that this is not a preferred method.
Is there another way to do this without writing my own web server. (I would like to use lambda).
If not, how can I configure my API Gateway method to support file upload to lambda?
You have a couple of options here:
Use API Gateway as an AWS Service Proxy to S3
Use API Gateway to invoke a Lambda function, which uses the AWS SDK to upload to S3
In either case, you will need to base64 encode the file content before calling API Gateway, and POST it in the request body.
We don't currently have any documentation on this exact use case but I would refer you to the S3 API and AWS SDK docs for more information. If you have any specific questions we'd be glad to help.
Thanks,
Ryan