I am making an app that uses S3 to store a lot of user media which I then display to the user at a later time. I am trying to figure out the best and most secure way to accomplish this.
I read that storing the media at S3 and then using the url from S3 to load media might be a bad choice because it can expose information that you might not want out in the open. Is it right to download all media to the sever from S3 before loading it on a page? that seems like it defeats the purpose of S3 in the first place if I have to keep downloading media from there in order to display it.
What is the best practice for accomplishing this?
I would appreciate a little advice.
Thanks!
There are many different ways to use S3. There isn't a single "best-practice".
Serve all content through web server:
In this scenario, S3 is used simply as a storage medium. Content is requested through the web server, which then downloads the content from S3 and forwards the contents to the client.
In this scenario, the client is not aware of the S3 bucket.
This does not defeat the purpose of S3 because the purpose of S3 (in this scenario) is content storage, not delivery.
Serve content from a public S3 bucket:
In this scenario, you setup your S3 bucket to serve up the content directly. In this case, all of the content is public, so direct linking to the content from the web app is used. All content in the S3 bucket can be downloaded by anyone.
The bucket can be referenced as bucket.s3-website.amazonaws.com, or under your own domain.
This scenario has the benefit that it offloads the delivery of the content from your web server.
Serve content from a private S3 bucket:
In this scenario, your bucket is private, but you still serve up the content directly. Using this system, you create expiring pre-signed URLs to protect the private content. The content is downloaded directly from S3, but not all content can be downloaded by everyone.
Like the previous scenario, this scenario has the benefit that it offloads the delivery of the content from your web server.
CloudFront:
You can use CloudFront in front of your app and/or S3 buckets to do any of the following:
cache the content, speeding up global delivery,
protect your web server, in conjunction with WAF
Final thoughts:
The setup you choose depends on your application.
Related
I have an application which is a static website builder.Users can create their websites and publish them to their custom domains.I am using Amazon S3 to host these sites and a proxy server nginx to route the requests to the S3 bucket hosting sites.
I am facing a load time issue.As S3 specifically is not associated with any region and the content being entirely HTML there shouldn't ideally be any delay.I have a few css and js files which are not too heavy.
What can be the optimization techniques for better performance? eg: Will setting headers ? or Leverage caching help? I have added an image of pingdom analysis for reference.
Also i cannot use cloudfront as when the user updates an image the edge locations have a delay of few minutes before the new image is reflected.It is not instant update,hence restricting the use for me. Any suggestions on improving it?
S3 HTTPS access from a different region is extremely slow especially TLS handshake. To solve the problem we invented Nginx S3 proxy which can be find over the web. S3 is the best as origin source but not as a transport endpoint.
By the way try to avoid your "folder" as a subdomain but specify only S3 regional(!) endpoint URL instead with the long version of endpoint URL, never use https://s3.amazonaws.com
One the good example that reduces number of DNS calls is the following below:
https://s3-eu-west-1.amazonaws.com/folder/file.jpg
Your S3 buckets are associated with a specific region that you can choose when you create them. They are not geographically distributed. Please see AWS doc about S3 regions: https://aws.amazon.com/s3/faqs/
As we can see in your screenshot, it looks like your bucket is located in Singapore (ap-southeast-1).
Are your clients located in Asia? If they are not, you should try to create buckets nearer, in order to reduce data access latency.
About cloudfront, it should be possible to use it if you invalide your objects, or just use new filenames for each modification, as tedder42 suggested.
I have a feature on my website where users can upload images. Users can see their own images but not others. The images are stored on Amazon S3 but uploaded and viewed on my website which is at a web hosting and not S3.
I have tried to show the pictures on my website through my private key when pictures are private at Amazon but failed.
Found this post: http://blog.learningtree.com/configuring-amazon-s3-to-serve-images which describes how to make the images/files more private even if they are set to public on S3. The site suggest to stop search engines with robots.txt file and only serves images to people who are coming from my domain to stop hot-linking.
Do you think this is enough if I make them public on S3 or should I think about something else?
You can also configure the images on S3 to be private, and then generate pre-signed URLs in your app. That way, you can include an expiry time within the link.
From Authenticating REST Requests in the S3 docs:
For example, if you want to enable a user to download your private data directly from S3, you can insert a pre-signed URL into a web page before giving it to your user.
People can then only use the generated URL for a certain time. If they come through your app, it will always generate a link for some time in the future (say, 15 minutes as an example). If people pass around the links to these images, these links auto-expire.
Most S3 SDKs have higher-level methods to pre-sign those URLs.
Relevant: How secure are presigned URLs in AWS S3? here on SO.
I am pretty new to aws. I want to store my video files in s3 bucket and host them on my website using cloudfront. Users should be able to download videos only after logging onto my website.
How do i go about implementing this? Since i am new to aws, a tutorial link would be very helpful. Thank You.
Also if you could suggest other cheaper but reliable CDNs for video files, it would be very helpful.
You can restrict your Amazon S3 content to accept requests only from Amaozn CloudFront and also use signed, temporary URLS for content delivery, thus serving private content through CloudFront.
See: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PrivateContent.html
I have a Python/Flask application that I've deployed in elastic beanstalk. I have been deploying updates via git aws.push, which includes my static js libraries, css, and images.
I now have about 1 GB of static content in the form of images. I want to serve that content from the same location as my application, that is, from the same place I was serving them before, in a /static/img/ folder. However, I obviously don't want to add the images to source control or deploy them with the git macro.
Ideally, I would like to connect to the instance where the files are hosted and upload them manually. However, I do not know how to do this. I have searched through the s3 associated with the elastic beanstalk app, but there is no sign of my app there, only a repository of zipped deployments.
I could create a new bucket and handle things that way, but I haven't been able to map a domain to a new bucket. Whenever I try to add a CNAME record to the bucket, it is rejected because "URL/IP cannot be added as a CNAME." In any case, the process that seems most intuitive is to manually put unversioned static content in place next to versioned, deployed code.
You're correct, this type of static content should not be part of your repository and certainly not stored on EC2 instance's volumes.
AWS' best practice for this use case would be to use S3 and directly link to S3 objects from your HTML code. S3 is a natively HTTP enabled object storage service.
In order to use S3 as web server, you must create a bucket on S3.
You can either use the S3 provided URL <bucket-name>.s3-website-<AWS-region>.amazonaws.com to link to your content from your web pages.
Or you can use your own domain name. In this case, your bucket name must be named after your domain name and you must enable "Website Hosting" option at the bucket level. This is required to let S3 know how to map HTTP requests to buckets.
A high level scenario is described here : http://docs.aws.amazon.com/gettingstarted/latest/swh/website-hosting-intro.html
And more details are provided by S3 documentation.
As an added benefit, storage in S3 costs less money than EBS storage.
I have a java application deployed at elastic beanstalk tomcat and the purpose of the application is to serve resources from S3 in zipped bundles. For instance I have 30 audio files that I zip up and return in the response.
I've used the getObject request from the AWS SDK, however its super slow, I assume it's requesting each object over the network. Is it possible to access the S3 resources directly? The bucket with my resources is located next to the beanstalk bucket.
Transfer from S3 to EC2 is fast, if they are in the same region.
If you still want faster (and reliable) delivery of files, consider keeping files pre-zipped on S3 and serve from S3 directly rather than going through your server. You can use signed URL scheme here, so that the bucket need not be public.
Next level is speed up is by keeping the S3 behind Cloudfront as an origin server. Here the files are cached in locations near your users. Serving Private Content through CloudFront