I am a mobile application developer. I use Google Cloud bucket to store 10-second videos and photos that I use in the application. Thousands of users use the application every day, and I want to use a CDN to ensure that the content of the application is delivered to users with minimum delays and maximum speed.
At the moment, I have only found an opportunity to create a bucket within one region to choose from: the USA, Europe, and Asia. How to create a universal bucket in Google Cloud Platform for storing application content, which would determine the user's location and serve content from the server closest to the user?
Thank you!
You can take advantage of Cloud CDN (fully) when you use an HTTP(S) Load Balancer since it's a global resource, meaning that your bucket traffic would be forwarded to a GCP Point of Presence (PoP) near all the clients worldwide.
If interested, here's how to do it, but keep in mind that you would (potentially) need to change your DNS value from current bucket to the new Load Balancer, and HTTP --> HTTPS redirection is not implemented by default (you can achieve an automatic redirect with this, but this is something you would need to setup separatedly).
On an additional note, depending on how many files (similar) are requested from your bucket, you will be charged less than it would be considered CDN traffic and not full GCP traffic.
So in short, your bucket data would be on your selected region, but CDN would be all around the globe meaning less latency, less price and your backend not being overwhelmed (this doesn't apply here since you are using buckets, but would apply for backend GCE instances).
Related
I'm new to AWS wolrd. My purpose is to find as soon as possible in case of problems using Elastic Load Balancer logs top ips from requests, if possible who they are or some inspection on it. I only found paid services. Does anyone know a free application or maybe a website that analyzes AWS ELB logs?
Completely free solution isn't available as I know. Btw, there are cheap solutions.
You can monitor your load balancer by "Access logs", "CloudWatch metrics", "Request tracing" and "CloudTrail logs".
I don't understand exactly what you want, but there are some possible solutions.
If you're afraid of being attacked and you need immediate protection (against security scans, DDoS etc), you can use AWS's own services. "AWS Shield Standard" is automatically included at no extra cost. Btw, "For added protection against DDoS attacks, AWS offers AWS Shield Advanced". https://docs.aws.amazon.com/shield/
WAF is also good against attacks. You can create rules, rule-actions etc. Sadly it's not completely free. It runs "pay-as-you-use" style. https://aws.amazon.com/waf/pricing/
you can store the access log in S3 and analyse it later, but this can be costly in the end (and it's not real time)
you can analyse your log records with Lambda function. In this case, you need to use some NoSQL or something to store states or logics. (Lambda and DynamoDB is "pay-as-you-use" style and cheap, but not for free)
Keep in mind that:
The load balancer and lambda also increments the corresponding CloudWatch metric (it's cheap, but not for free)
You will pay for the outgoing data transfer. I mean from AWS to internet 1TB/month/account is always free (through CloudFront): https://aws.amazon.com/free/
you should use AWS's own services if you want a cheap and good solution
Elastic Load Balancing provides access logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses.
But keep in mind that access logging is an optional feature of Elastic Load Balancing that is disabled by default. After you enable access logging for your load balancer, Elastic Load Balancing captures the logs and stores them in the Amazon S3 bucket that you specify as compressed files. You can disable access logging at any time.
There are many complex and paid application that returns information regard access log but i advise you a simple, easy to use website that i use when i want to see top requester on our load balancer.
Website is https://vegalog.net
You shoud only upload your log file taken from S3 bucket and it returns to you a report with top requester, who they are (using whois function), response time and other useful informations.
I'm reading Computer Networking A Top-Down Approach, in a section about CDN(Content Distribution Networks), it chooses Netflix as a case:
As shown in Figure, this Web site (and its associated backend
databases) run entirely on Amazon servers in the Amazon cloud.
Additionally, the Amazon cloud handles the following critical
functions:
Content ingestion. Before Netflix can distribute a movie to its
customers, it must first ingest and process the movie. Netflix
receives studio master versions of movies and uploads them to hosts in
the Amazon cloud.
Content processing. The machines in the Amazon cloud
create many different formats for each movie, suitable for a diverse
array of client video players running on desktop computers,
smartphones, and game consoles connected to televisions. A different
version is created for each of these formats and at multiple bit
rates, allowing for adaptive streaming over HTTP using DASH.
Uploading versions to its CDN. Once all of the versions of a movie have been
created, the hosts in the Amazon cloud upload the versions to its CDN.
I can't understand what's difference between Amazon cloud and its CDN.
Yes, the Amazon CloudFront CDN is part of the 'AWS Cloud', but it distinct enough to considered somewhat separate.
From Global Infrastructure Regions & AZs, AWS has (as at Oct 2021):
25 Regions (plus 8 more announced)
218+ Edge Locations and 12 Regional Edge Caches
If you want to store data in Amazon S3, or create virtual computers in Amazon EC2, they operate in Data Centers in the 25 Regions around the world.
Amazon CloudFront (which is their Content Distribution Network) keeps caches in those 218+ locations. There are more Edge Locations than Regions so that data can be kept 'closer' to users. In fact, there are sometimes multiple Edge Locations in the same city because they are located in facilities run by major network providers and ISPs, thereby reducing the latency when serving users on those major networks.
Amazon CloudFront is a 'pull' system. An Edge Location only caches content when it is requested by a user close to that Edge Location. It works like this:
A user tries to access a website that is served by CloudFront
DNS resolution of the domain name automatically redirects users to the 'closest' location (or, more accurately, the location with the lowest latency)
If the requested data is already present in the cache, it is immediately returned.
Otherwise, a request is sent to the Origin. Any data that is returned is then stored in the cache and returned to the user
Any future request to that Edge Location will then be able to serve the cached content until it expires
So, your quotation is not quite accurate. It is not possible to 'upload' content to Amazon CloudFront. Instead, it is 'pulled' into the cache locations when requested.
Also, Netflix uses its own cache network by installing Open Connect servers in the data centers of many large network providers and ISPs, thereby avoiding network costs and latency.
I have website (EC2, RDS, VPC, S3) located in EU (Ireland) and I want to make it more accessible for users from America and Asia.
Should I create new instances (EC2, RDS, VPC, S3) in new regions? Or there is another way how to do that?
If I will have more EC2 instances, how should I deploy updates for every instance?
What is the best way to make AWS website light and accessible with small latency from every corner of the world?
Should I create new instances (EC2, RDS, VPC, S3) in new regions?
If you take budget considerations out of the picture then creating instances in each AZ around the world and spreading geographic traffic to them would be a great consideration.
Or there is another way how to do that?
Perhaps the easiest way both from implementation and maintainability as well as budget considerations would be to implement a geographic edge cache like Akamai, CloudFlare, etc.
Akamai is horrendously expensive, but CloudFlare has some free and very cheap plans.
Using an edge cache means that static cached content can be served to your clients from the nearest global edge points to them, without requiring your AWS infrastructure to be optimised for regions.
For example - if you request your home page from Ireland, it may be served from an Irish edge cache location, whereas if I request it from New Zealand, it may be served from an Australasian edge cache location - this doesn't add any complexity to your AWS set up.
In the scenario where a cached version of your page doesn't exist in CloudFlare (for example), it will hit your AWS origin server for the result. This will result in geographic performance variation, but you trade that off against the cost of implementing EC2 instances in each region and the reduced number of hits that actually reach your infrastructure with the cache in place.
If I will have more EC2 instances, how should I deploy updates for every instance?
This largely depends on the complexity of your web application.
For more simple applications you can use Elastic Beanstalk to easily deploy updates to all of your EC2 instances and manage your auto-scaling.
For more complex arrangements you might choose to use a combination of CodeCommit, CodePipeline and CodeDeploy to achieve the same thing.
Thirdly, there's always the option that you could construct your own deployment tool using a combination of custom scripts and AWS API calls. Or use a tool that has already been created for this purpose.
What is the best way to make AWS website light and accessible with small latency from every corner of the world?
This is a pretty broad and complicated question.
My suggestions would be to make use of lazy loading wherever possible, cache everything wherever you can, tweak your web server configuration within an inch of its life (and use things like Varnish if you're on nginx), optimise all your media assets as much as possible, etc.
For media assets you could use a CDN (like S3 or CloudFront) to serve requests instead of storing them on EC2 instances.
By far the most important thing you could do for this though would be to put in an edge cache (discussed earlier). If you do this, your AWS performance is much less of a concern.
Im developing global mobile service communicating with back end server ( S3 - file server , EC2 - application server)
But i don't know how many s3 and ec2 are needed and where i should launch these.
So i'd like to know about below
Im planning to mount S3 in Oregon. As you know, CloudFront is the good solution for getting image quickly but the problem i wanna solve is uploading. I thought 2 solutions. The first solution it that using Put method to CloudFront, upload file to S3 through CloudFront. The second solution is mounting several S3 in different regions. Which is the better solution?
Now i am developing application server in only one EC2. I might have to mount several EC2s for global service. but i don't know how to make end users to connect to specific ec2 of several EC2s. Can you explain me?
thanks
I think your understanding of S3, is slightly off.
You wouldn't and shouldn't need to create "Geo"-specific S3 buckets for the purposes you are describing.
If you are using the service for image delivery over pure HTTP, then you can create the bucket anywhere, and then use a Amazon Cloudfront Distribution as the "frontend" whihc will give you approximately 40 edge locations around the world for your Geo-optimizations.
The more relevant edge location will be used for each user around the world, and they will then request the image from your S3 bucket and store it based on your meta settings. (Typically, from my experience, it's about every 24 hours for low serving traffic websites even when you set an Expire age of months/years.
You also don't "mount" S3. You just create a bucket, and you shouldn't ever want to create multiple buckets which store the same data.
.........
For your sercond question, regarding creating a "global service" for EC2, what are you hopeing to actually achieve.
The web is naturally global. Are your users going to be frett over an additional 200ms latency?
You haven't really descrived what your service will do, but one approach, would be to do all of your computing in Oregon, and then just create cache servers, such as Varnish in different regions. You can use Route53 for the routing, and you can also take advantage of ELB.
My recommendation would be to stop what you are doing, and launch everything from Oregon. The "web" is already "global" and I don't think you are going to need to worry about issues such as this until you hit scale. At which point, I'm going to assume you can just hire someone to solve this problem for you. It sounds like you have the budget for it...
First of all, I am pretty new to AWS, so my question might seem very amateur.
I am a developing a web application which needs to available globally and currently am hosting it on amazon. Since the application is still under development, i have set it up in the Singapore region. However, when i test the application, i get good response times from locations on the the east side of the globe(~50ms). However, when i test the response times from the US, it's ~550ms. So we decided to have 2 instances one in Singapore and one in the US. But i'm not able to figure out a way to handle data replication and load balancing across regions. Elastic Beanstalk only allows me to do this in a particular region. Can somebody please explain how i can achieve global availability for my web app. The following are the services i currently use.
1. Amazon EC2
2. Amazon S3
I need both database replication and S3 file replication. Also it would be great if there was a way where i just need to deploy my application on one place and the changes are reflected across all the instances we would have on the globe.
Before you spend a lot of time and money setting up redundant servers in different regions, you may want to make sure that you can't get the performance improvement you need simply by implementing AWS Cloudfront:
Amazon CloudFront employs a network of edge locations that cache
copies of popular files close to your viewers. Amazon CloudFront
ensures that end-user requests are served by the closest edge
location. As a result, requests travel shorter distances to request
objects, improving performance. For files not cached at the edge
locations, Amazon CloudFront keeps persistent connections with your
origin servers so that those files can be fetched from the origin
servers as quickly as possible. Finally, Amazon CloudFront uses
additional optimizations – e.g. wider TCP initial congestion window –
to provide higher performance while delivering your content to
viewers.
http://aws.amazon.com/cloudfront/faqs/
The nice thing is, you can set this up and test it out in very little time and for very little money. Obviously this won't solve all performance problems, especially if you app is performance bound at the database, but this is a good way of taking care of that 'low hanging fruit' when trying to speed up your website in diverse locations around the world.