Our media image files are stored under
http://s3.ap-northeast-2.amazonaws.com/{bucket_nmae}/media/
I want to change the url to
http://static.example.com/media/ and serve them through cloudflare / cloudfront if possible
I've seen tutorials which describe the steps for using s3 as your endpoint or CDN as your endpoint. (https://ruddra.com/posts/aws-boto3-useful-functions/)
But I haven't found one that describes the steps to move from s3 to CDN .
Specifically,
Do I need to move files from s3 to CDN manually?
I think image field itself doesn't have URL attached to it, and once we move (or connect) s3 image to CDN, I believe one can use http://static.example.com instead of http:///s3.ap-northeast-2.amazonaws.com/{bucket_name}/
What about the image urls stored in the database..
For instance, when you upload image as a part of a posting, the posting html might have the full image url. These will require DB data migrations.. I belive
To create your custom domain for a bucket you will need:
create a distribution in CloudFront and set the origin as your bucket
create a Record Set in Route 53 and point your CloudFront endpoint created in CNAME record.
That is it.
Related
Setup:
We are running a E-commerce website consists of Cloudfront-->ALB-->EC2. we are serving the images from S3 via cloudfront behaviour.
Issue:
Our admin URL is like example.com/admin. We are uploading product images via admin panel as a zip file that goes via cloudfront.Each zip file size around 100MB-150MB consists of around 100 images. While uploading the zip file we are facing 502 gateway error from cloudfront since it took more than 30sec, which is default time out value for cloudfront.
Expected solution:
Is there a way we can skip the cloudfront for only uploading images?
Is there any alternate way increasing timeout value for cloudfront??
Note: Any recommended solutions are highly appreciated
CloudFront is a CDN service to help you speed up your services by caching your static files in edge location. So it won't help you in uploading side
In my opinion, for the uploading images feature, you should use the AWS SDK to connect directly with S3.
If you want to upload files directly to s3 from the client, I can highly suggest using s3 presigned URLs.
You create an endpoint in your API to create the presigned URL for a certain object (myUpload.zip), pass it back to the client and use that URL to do the upload. It's safe, and you won't have to expose any credentials for uploading. Make sure to set the expiration time to a reasonable time (one hour).
More on presigned URLs's here https://aws.amazon.com/blogs/developer/generate-presigned-url-modular-aws-sdk-javascript/
I am using AWS S3 bucket for storing and displaying the user upload image files.
My website name is example.net and as I want to use cloudflare CDN and show the image url path as images.example.net (as I want to mask the S3 url).
I created a S3 bucket with name images.example.net and mapped the CNAME images.example.net.s3.us-east-1.amazonaws.com as images.
So I am able to access the images using url images.example.net/myPic.jpg
Recently I got a new domain example.com (only the extension is different), and now I want to use the same S3 bucket example.net to store the files and the url needs to shown as images.example.com/myPic.jpg
If I create a new S3 bucket images.example.com , then I can do it, but I want to use the existing one only.
Is there any method to configure in cloudflare for that ?
Can I do it without creating a new bucket named images.example.com ?
If you're using virtual hosting then you're limited to a single domain per bucket by using the domain as the bucket name.
The only options available to you are:
Create a second S3 bucket with your other domain name as the CNAME
Create a CloudFront distribution in front of your S3 bucket and specify both domains as alternative domain names. Then on CloudFlare use the CloudFront CNAME. Disable all caching to allow CloudFlare to handle all the CDN functionality.
Obviously neither are ideal for your scenario, however these are the options available at this time.
I'm trying to serve static web content (HTML, CSS, and JS files) from S3 buckets. I know I can go to the bucket's properties tab and choose the item Use this bucket to host a website from the Static website hosting box. And I'm sure this step will still be part of the solution I'm looking for but it won't be all.
Here's what I'm trying to accomplish:
Deploying the same content to multiple regions and based on availability and/or latency, provide the service to the client.
As for the API Gateway, I know how to do this. I should create the same API Gateway (alongside underlying lambda functions) and Custom Domain Names in all the regions. And then creating the same domain on Route 53 (of type CNAME) and choose Latency as Routing Policy. One can also set up a Health Check for the Record Set so availability of the API Gateway and lambda functions are checked periodically.
Now I want to do the same for the S3 bucket and my static content. i.e. I want to deploy the same content to different regions and somehow make Route 53 to route the request to the closest available bucket. Previously, I was using CloudFront but it seems to me in this setup, I can only introduce one bucket.
Does anyone know how can I serve my static content from multiple buckets? If you are going to suggest CouldFront, please tell me how you plan to use multiple buckets.
You can generate a certificate, setup a CloudFront distribution to grab the content from your bucket and then point your domain to your distribution using Route53. You get free https and you can also add several S3 buckets as origins for your distribution.
From AWS Docs:
After you configure CloudFront to deliver your content, here's what happens when users request your objects:
1. A user accesses your website or application and requests one or more objects, such as an image file and an HTML file.
2. DNS routes the request to the CloudFront edge location that can best serve the request—typically the nearest CloudFront edge location in terms of latency—and routes the request to that edge location.
3. In the edge location, CloudFront checks its cache for the requested files. If the files are in the cache, CloudFront returns them to the user. If the files are not in the cache, it does the following:
3a. CloudFront compares the request with the specifications in your distribution and forwards the request for the files to the applicable origin server for the corresponding file type—for example, to your Amazon S3 bucket for image files and to your HTTP server for the HTML files.
3b. The origin servers send the files back to the CloudFront edge location.
3c. As soon as the first byte arrives from the origin, CloudFront begins to forward the files to the user. CloudFront also adds the files to the cache in the edge location for the next time someone requests those files.
P.D. Keep in mind this is for static content only!
This is possible with CloudFront using Lambda#Edge to change origin based on answer from Route 53.
Please refer this blog for a sample Lambda#Edge code to do this -
https://aws.amazon.com/blogs/apn/using-amazon-cloudfront-with-multi-region-amazon-s3-origins/
I've been using AWS_CLOUDFRONT_DOMAIN = 'media.example.com'
One consequence is that I have database records of image urls which have the above domain name enbeded such as http://media.example.com/media/blog_image_web/2016-07-01/702ef913-ddeb-4880-a715-397ce7b9efd7.jpg
Now I'd like to stop using cloudfront and just want to server from S3.
Problem is S3 url looks like http://my.bucketname.s3.amazonaws.com/media/blog_image_web/2016-07-01/702ef913-ddeb-4880-a715-397ce7b9efd7.jpg for the same image file..
I can update the database by replacing
http://media.example.com/ to http://my.bucketname.s3.amazonaws.com/
However, it would be much better if I can just keep using http://media.example.com/ whether I use cloudfront or not (or I could move my bucket later).
How can I use a custom domain to access s3 objects?
You can assign a custom domain to point to an Amazon S3 bucket.
The basic steps are:
Create an S3 bucket with a name equal to the domain name (eg media.example.com)
Activate Static Website Hosting on the bucket
Create an A record in Route 53 for media.example.com with Alias = YES and point it to your S3 website endpoint
See documentation: Example: Setting Up a Static Website Using a Custom Domain
We want to host images on our application as fast as possible. As we already have an AWS setup we prefer to host our images on S3 buckets (but are open for alternatives).
The challenge is routing the request to the closest S3 bucket.
Right now we use Amazon Route 53 with geolocation routing policy to the closes EC2 instance wich redirects to the respective bucket. We find this inefficent as the request goes:
origin->DNS->EC2->S3 and would prefer
origin->DNS->S3. Is it possible to bind two static website S3 buckets to the same domain where request are routed based on Geolocation?
Ps: We have looked into cloudfront, but since many of the images are dynamic and are only viewed once we would like the origin to be as close to the user as possible.
It's not possible to do this.
In order for an S3 bucket to serve files as a static website, the bucket name must match the domain that is being browsed. Due to this restriction, it's not possible to have more than one bucket serve files for the same domain because you cannot create more than one bucket with the same name, even in different regions.
CloudFront can be used to serve files from S3 buckets, and those S3 buckets don't need to have their names match the domain. So at first glance, this could be a workaround. However, CloudFront does not allow you to create more than one distribution for the same domain.
So unfortunately, as of this writing, geolocating is not possible from S3 buckets.
Edit for a deeper explanation:
Whether the DNS entry for your domain is a CNAME, an A record, or an ALIAS is irrelevant. The limitation is on the S3 side and has nothing to do with DNS.
A CNAME record will resolve example.com to s3.amazonaws.com to x.x.x.x and the connection will be made to S3. But your browser will still send example.com in the Host header.
When S3 serves files for webpages, it uses the Host header in the HTTP request to determine from which bucket the files should be served. This is because there is a single HTTP endpoint for S3. So, just like when your own web server is hosting multiple websites from the same server, it uses the Host header to determine which website you actually want.
Once S3 has the Host that you want, it compares it against the buckets available. It decided that the bucket name would be used to match against the Host header.
So after a lot of research we did not find an answer to the problem. We did however update our setup. The scenario is that a user clicks a button and will view some images in an IOS app. The request when the user pushes the button is geo rerouted to the nearest EC2 instance for faster performance. Instead of returning the same imagelinks in EU and US we updated it so when clicking in US you get links to an American S3 bucket and the same for Europe. We also put up two cloud front distributions, one in front of each S3 bucket, to increase speed.