What I need:
- load video from client
- cut this video on chunks by timepoints
- store thuis chunks
- provide access to this video chunks for web-users
Could you please give some advices how to properly build this process using AWS infrastructure?
You have a very broad question, so you cannot expect a very detailed answer. But lets start at least with basic puzzle pieces.
AWS may provide you infrastructure and services to support your case.
load video from client
Commonly the uploads are to be stored in an S3 bucket.
cut this video on chunks by timepoints
Once the video is uploaded, you may use the Elastic Transcoder service or any application on a virtual machine (AWS EC2, AWS Batch,..) to process the uploaded video files. You can use the Elastic Transcoder to generate clips (chunks)
store thuis chunks - provide access to this video chunks for web-users
The chunks can be stored in S3 again and you can make a web app to reference (give access) to the stored chunks
This is at least basic overview, but based on your question it may be a good start
Related
I have prepared a CNN and would like to deploy it on AWS to do some image classification. However, the images are stored on an other server than Amazon S3.
Do I have to load all the images on S3 prior to calling the endpoint to make inference ? Do AWS handle like a "cache memory" so I can get images from that server without bringing them on S3 ?
On an other note, is there any alternative way to make classification of large amount of images ? The output should be a Json file. I'm quite lost with all the AWS features.
Thank you for your help !
At the moment, Batch Transform is limited to S3 as an input source.
To better suggest an Inference method can you please state the size of the input data and latency requirements?
SageMaker also offers Asynchronous Inference that allows you to point to an S3 path(input) and select an Instances and count with an optional scaling policy to scale down to zero to save costs. The maximum size of input for Async is 1GB.
I want to extract one frame or screenshot of a video stored in s3 at specific time, what can i use to make it ?
Lambda functions
Using the SDK
Amazon Elastic Transcoder has the ability to create videos from sources files. For example, it can stitch together multiple videos, or extract a portion of video(s).
Elastic Transcoder also has the ability to generate thumbnails of videos that it is processing.
Thus, you should be able to:
Create a job in Elastic Transcoder to create a very short-duration video from the desired time in the source video
Configure it to output a thumbnail of the new video to Amazon S3
You can then dispose of the video (configure S3 to delete it after a day) and just use the thumbnail.
Please note that Elastic Transcoder works asynchronously, so you would create a Job to trigger the above activities, then come back later to retrieve the results.
The benefit of the above method is that there is no need to download or process the video file on your own Amazon EC2 instance. It is all done within Elastic Transcoder.
The AWS SDK does not have an API that extracts pictures from a video. You can use AWS to analyze videos - such as the Amazon Rekognition service. For example:
Creating AWS video analyzer applications using the AWS SDK for Java
You can use the Amazon Rekognition to detect faces, objects, and text in videos. For example, this example detects text in a video:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/rekognition/src/main/java/com/example/rekognition/VideoDetectText.java
The Amazon S3 API has many operations, but extracting a pic from a video is not one of them. You can get an inputstream of an object located in a bucket.
To extract a pic from a video, you would need to use a 3rd party API.
I have an application which streams video like(NetFlix, Youtube).
I am trying to host it in the AWS platform. I have found two different options with this:
first one is store video files in s3.
the second one is store video files in AWS MediaStore.
In my existing platform, I have a problem with downloading video through IDM by end users.
So, I have to prevent downloading the video from IDM.
How can I do this in the AWS platform? Which AWS service will suit my case of preventing downloading?
Please take note of data-out charge when you use AWS as the primary mean to serve your video streams. Personally I found It prohibitively expensive to use AWS's service to serve your video
Netflix for example use S3 as a part of main storage for their video streams.
To the question of which service you can use to hide direct link / download link from AWS. Currently there is no service provided natively by AWS for that purpose
My client has a service which stores a lot of files, like video or sound files. The service works well, however looks like the long-time file storing is quite a challenge, and we would like to use AWS for storing these files.
The problem is the following, the client wants to use AWS kinesis for transferring every file from our servers to AWS. Is this possible? Can we transfer files using that service? There's a lot of video files, and we got more and more every day. And every files is relatively big.
We would also like to save some detail of the files, possibly into dynamoDB, we could use Lambda functions for that.
The most important thing, that we need a reliable data transfer option.
KInesis would not be the right tool to upload files, unless they were all very small - and most videos would almost certainly be over the 1MB record size limit:
The maximum size of a data blob (the data payload before
Base64-encoding) within one record is 1 megabyte (MB).
https://aws.amazon.com/kinesis/streams/faqs/
Use S3 with multi-part upload using one of the SDK's. Objects you won't be accessing for 90+ days can be moved to Glacier.
Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.
Amazon Web Services. Amazon Simple Storage Service (S3) Developer Guide (Kindle Locations 4302-4306). Amazon Web Services, Inc.. Kindle Edition.
To further optimize file upload speed, use transfer acceleration:
Amazon S3 Transfer Acceleration enables fast, easy, and secure transfers of files over long distances between your client and an S3 bucket. Transfer Acceleration takes advantage of Amazon CloudFront’s globally distributed edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path.
Amazon Web Services. Amazon Simple Storage Service (S3) Developer Guide (Kindle Locations 2060-2062). Amazon Web Services, Inc.. Kindle Edition.
Kinesis launched a new service "Kinesis Video Streams" - https://aws.amazon.com/kinesis/video-streams/ which may be helpful to move large amount of data.
Im developing a mobile app that will use AWS for its backend services. In the app I need to upload video files to S3 on a frequent basis, and I'm wondering what the recommended architecture would look like to make this scalable and efficient. Traffic could be high, and file sizes could be large.
-On one hand, I could upload directly to S3 using the S3 API on the client side. This would be the easiest option, but Im not sure of the negative implications associated with it.
-The other way to do it would be to go through an EC2 instance and handle the request using some PHP scripts and upload from there.
So my question is... Are these two options equal, or are there major drawbacks to one of them opposed to another? I will already have EC2 instances configured for database access if that makes any difference in how you approach the question.
I will recommend using "upload directly to S3 using the S3 API on the client side" as you can speed up the upload process by using AWS S3 part upload as your video files are going to large.
The second method will put extra CPU usage load on your EC2 instance as the script processing and upload to S3 will utilize CPU for the process.