I have to upload video files into an S3 bucket from my React web application. I am currently developing a simple react application and from this application, I am trying to upload video files into an S3 bucket so I have decided two approaches for implementing the uploading part.
1) Amazon EC2 instance: From the front-end, I am hitting the API and the server is running in the Amazon EC2 instance. So I can upload the files into S3 bucket from the ec2 instance.
2) Amazon API Gateway + Lambda: I am directly sending the local files into an S3 bucket through API + Lambda function by calling the https URL with data.
But I am not happy with these two methods because both are more costly. I have to upload files into an S3 bucket, and the files are more than 200MB. I don't know I can optimize this uploading process. Video uploading part is necessary for my application and I should be very careful to do this part and also I have to increase the performance and cost-effective.
If someone knows any solution please share with me, I will be very helpful for me to continue my process.
Thanks in advance.
you can directly upload files from your react app to s3 using aws javascript sdk and cognito identity pools and for the optimization part you can use AWS multipart upload capability to upload file in multiple parts I'm providing links to read about it further
AWS javascript upload image example
cognito identity pools
multipart upload to S3
also consider a look at aws managed upload made for javascript sdk
aws managed upload javascript
In order to bypass the EC2, you can use a pre-authenticated POST request to directly upload you content from the browser to the S3 bucket.
Related
I’m hosting videos on my site via s3 buckets. I have a video that keeps buffering. The video is 4K and 6.5GB. Smaller videos shot in a lower resolution do not buffer. I’m having a hard time deciding whether it’s the video’s size in GB’s or 4K resolution that’s making it buffer. Anyone knows what makes a video buffer from a s3 bucket? Is it the size of the video or the resolution of the video? Also, does any know how to stop video buffering. Yes, I’ve already tried using cloud front but the same result.
Resolution
For large files, Amazon S3 might separate the file into multiple uploads to maximize the upload speed. The Amazon S3 console might time out during large uploads because of session timeouts. Instead of using the Amazon S3 console, try uploading the file using the AWS Command Line Interface (AWS CLI) or an AWS SDK.
Note: If you use the Amazon S3 console, the maximum file size for uploads is 160 GB. To upload a file that is larger than 160 GB, use the AWS CLI, AWS SDK, or Amazon S3 REST API.
AWS CLI
First, install and configure the AWS CLI. Be sure to configure the AWS CLI with the credentials of an AWS Identity and Access Management (IAM) user or role. The IAM user or role must have the correct permissions to access Amazon S3.
Important: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.
To upload a large file, run the cp command:
aws s3 cp cat.png s3://docexamplebucket
Note: The file must be in the same directory that you're running the command from.
When you run a high-level (aws s3) command such as aws s3 cp, Amazon S3 automatically performs a multipart upload for large objects. In a multipart upload, a large file is split into multiple parts and uploaded separately to Amazon S3. After all the parts are uploaded, Amazon S3 combines the parts into a single file. A multipart upload can result in faster uploads and lower chances of failure with large files.
For more information on multipart uploads, see How do I use the AWS CLI to perform a multipart upload of a file to Amazon S3?
AWS SDK
For a programmable approach to uploading large files, consider using an AWS SDK, such as the AWS SDK for Java. For an example operation, see Upload an object using the AWS SDK for Java.
Note: For a full list of AWS SDKs and programming toolkits for developing and managing applications, see Tools to build on AWS.
for details:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-large-file-uploads/
Setup:
We are running a E-commerce website consists of Cloudfront-->ALB-->EC2. we are serving the images from S3 via cloudfront behaviour.
Issue:
Our admin URL is like example.com/admin. We are uploading product images via admin panel as a zip file that goes via cloudfront.Each zip file size around 100MB-150MB consists of around 100 images. While uploading the zip file we are facing 502 gateway error from cloudfront since it took more than 30sec, which is default time out value for cloudfront.
Expected solution:
Is there a way we can skip the cloudfront for only uploading images?
Is there any alternate way increasing timeout value for cloudfront??
Note: Any recommended solutions are highly appreciated
CloudFront is a CDN service to help you speed up your services by caching your static files in edge location. So it won't help you in uploading side
In my opinion, for the uploading images feature, you should use the AWS SDK to connect directly with S3.
If you want to upload files directly to s3 from the client, I can highly suggest using s3 presigned URLs.
You create an endpoint in your API to create the presigned URL for a certain object (myUpload.zip), pass it back to the client and use that URL to do the upload. It's safe, and you won't have to expose any credentials for uploading. Make sure to set the expiration time to a reasonable time (one hour).
More on presigned URLs's here https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/
I can't find some information about Amazon S3, hope you will help me. When is a file available for user to download, after the POST upload? I mean some small JSON file that doesn't require much processing. Is it available to download immediately after uploading? Or maybe amazon s3 works in some sessions and it always takes a few hours?
According to the doc,
Amazon S3 provides strong read-after-write consistency for PUTs and DELETEs of objects in your Amazon S3 bucket in all AWS Regions.
This means that your objects are available to download immediately after it's uploaded.
An object that is uploaded to an Amazon S3 bucket is available right away. There is no time period that you have to wait. That means if you are writing a client app that uses these objects, you can access them as soon as they are uploaded.
In case anyone is wondering how to programmatically interact with objects located in an Amazon S3 bucket through code, here is an example of uploading and reading objects in an Amazon S3 bucket from a client web app....
Creating an example AWS photo analyzer application using the AWS SDK for Java
I wanted to know if I can accomplish my aim of uploading files or large files from my mobile application to my s3 bucket without login into my amazon console. If there is no way to do this, then. How can I upload files once logged in without the annoying web view? Se what others tried.
Should I upload files to Amazon S3 from mobile devices or from my server? Thanks.
What I need to achieve:
Upload files to S3 with no hassles.
Download files when needed with no hassles.
Able to upload large files on S3.
I really don't want to mess with access key and authentication key. Well if I can at least hide them form onlookers that will be fine.
If you can share all the possible file upload implementation in the mobile application scenario that will be great thanks.
You can use SDK for AWS. It must be an application developed in some programming language available for AWS and be able to access the credentials to connect to S3 and work like an interface between the app and the S3 bucket.
All AWS services can be accessed programmatically via an API. In fact, this is the recommended method for using AWS services.
Mobile applications can certainly communicate directly with Amazon S3.
You will want to secure data stored in S3 so that only authorized and authenticated users can access it. This can be accomplished by having the mobile application authenticate with your own back-end service that handles authentication. Alternatively, you could use Amazon Cognito to handle authentication. This way, your app will handle authentication with AWS and users will never see AWS-specific credentials.
I have a java application deployed at elastic beanstalk tomcat and the purpose of the application is to serve resources from S3 in zipped bundles. For instance I have 30 audio files that I zip up and return in the response.
I've used the getObject request from the AWS SDK, however its super slow, I assume it's requesting each object over the network. Is it possible to access the S3 resources directly? The bucket with my resources is located next to the beanstalk bucket.
Transfer from S3 to EC2 is fast, if they are in the same region.
If you still want faster (and reliable) delivery of files, consider keeping files pre-zipped on S3 and serve from S3 directly rather than going through your server. You can use signed URL scheme here, so that the bucket need not be public.
Next level is speed up is by keeping the S3 behind Cloudfront as an origin server. Here the files are cached in locations near your users. Serving Private Content through CloudFront