when I make an unauthenticated (public) Cloud Run endpoint to host an API, what are my options to protect this endpoint from malicious users making billions of HTTP requests?
For $10 you can launch a Layer 7 HTTP flood attack that can send 250k requests per second. Let's assume your Cloud Run endpoints scale up and all requests are handled. For invocations alone, you will pay $360,-/hour (at $0.40 per million requests).
Note that there is a concurrency limit and a max instance limit that you might hit if the attack is not distributed over multiple Cloud Run endpoints. What other controls do I have?
As I understand, the usual defenses with Cloud Armor and Cloud CDN are bound to the Global Load Balancer, which is unavailable for Cloud Run, but is available for Cloud Run on GKE.
For unauthenticated invocations to a Cloud Run service with an IAM Cloud Run Invoker role set to the allUsers member type, I would expect the answer to be the same as those provided here - https://stackoverflow.com/a/49953862/7911479
specifically:
Cloud Functions sits behind the Google Front End which mitigates and absorbs many Layer 4 and below attacks, such as SYN floods, IP fragment floods, port exhaustion, etc.
It would certainly be great to get a clear Y/N answer on Cloud Armor support.
[Edit]: I have been thinking on this quite a lot and have come to the following conclusion:
if you expect you are likely to become a victim of an attack of this type then I would monitor your regular load/peak and set your account's ability to scale just above that load. Monitoring will allow you to increase this as your regular traffic grows over time. It appears to be the only good way. Yes, your service will be down once you reach your account limits, but that seems preferable in the scenario where you are the target.
An idea which I am yet to try is a protected route with Firebase Authentication and anonymous authentication.
Related
I'm considering a cloud run for web hosting rather than a complex compute engine.
I just want to make an api with node.js. I heard that automatic load balancing is also available. If so, is there any problem with concurrent traffic of 1 million people without any configuration? (The database server is somewhere else (which is serverless like cockroachDB)
Or Do I have to configure various complicated settings like aws ec2 or gce?
For such traffic, out of the box configuration must be fine tuned.
Firstly, the concurrency parameter on Cloud Run. This parameter indicate how many concurrent request can be handle per instance. It's 80 the default value, and you can set up to 1000 concurrent requests per instance.
Of course, if you handle 1000 concurrent request per instance (or less) you should require more CPU and Memory. You can also play with those parameters
You also have to change the max instance limit. By default, you are limited to 1000.
If you set 1000 concurrent requests and 1000 instances, you can handle 1 million of concurrent request.
However, you don't have a lot of margins, or your instance with 1000 concurrent requests can be struggle even with max CPU and memory.
You can request more than 1000 instances with a quota increase request.
You can also optimise differently, especially if your 1 million users aren't in the same country/Google Cloud Region. if so, you can deploy a HTTPS load balancer in front of your cloud run service and deploy it in all the region of your users. (The Cloud Run services deployed in different regions must have the same name).
Like that, it's not only one service that will have to absorb 1 million of users, but several, in different regions. In addition, the HTTPS load balancer route the request to the closest region and therefore your optimize the latency, and reduce the egress/cross region traffic.
I want to know the limit of requests per second for Load Balancer on Google Cloud Platform. I didn't found this information on documentation.
My project is a static website hosted on Storage Bucket behind the Load Balancer and CDN active,
This website will receive a campaign in a Television channel and the estimative is that 100k requests per second for 5 minutes.
Could anyone help me with this information? Its necessary to ask Support for pre-warmup the load balancer before the campaign starts?
From the front page of GCP Load Balancing:
https://cloud.google.com/load-balancing/
Cloud Load Balancing is built on the same frontend-serving
infrastructure that powers Google. It supports 1 million+ queries per
second with consistent high performance and low latency. Traffic
enters Cloud Load Balancing through 80+ distinct global load balancing
locations, maximizing the distance traveled on Google's fast private
network backbone.
This seems to say that 1 million+ request per second is fully supported.
However, with all that said ... I wouldn't wait for "the day" before testing. See if you can't practice a suitable load. Given that this sounds like a finite event with high visibility (television), I'm sure you don't want to wait for the event only to find out something was wrong in the setup or theory. From the perspective of "is 100K request per second through a load balancer" ... the answer appears to be yes.
If you (or you asking on behalf of) a GCP consumer, Google has Technical Account Managers associated with accounts that can be brought into the planning loop ... especially if there are questions on "can we do this". One should always be cautious of sudden high volume needs of GCP resources. Again, through a Technical Account Manager, it does no harm to pre-warn Google of large resource requests. For example, if you said that you needed an extra 5000 Compute Engines, you may be constrained on what regions are available to you given a finite existing capacity. Google, just like other public cloud providers, has to schedule and balance resources in its regions. Timing is also very important. If you need a sudden burst of resources and the time that you need them happens to coincide with some event such as Black Friday (US) or Singles Day (China) special preparation may be needed.
Im going to launch an app and Im worried if my competitors would just kill me by draining my Amazon AWS resources by using a botnet to send gibberish http requests to my Amazon AWS Account. I only got a few thousand dollars and I can not afford to be slaughtered like that.
In what other ways my competitors or haters could drain my server resources to drain my bank balance and how to prevent it?
please help. Im in very stressful situation where I cant get any answer for this question. Any suggestion is welcome.
Thanks.
As pointed by #morras, AWS Shield + WAF is good combination to protect your resources from spam requests. Since you have not given your architecture about what aws services you are actually using, I am trying to answer based on general term.
In AWS Shield there are two types
Standard - Automated mitigation techniques are built-into AWS Shield Standard, giving you protection against common, most frequently occurring infrastructure attacks. If you have technical expertise to create rules based on your request, you can go with this.
Advanced - AWS WAF comes free with this, and you will have 24x7 access to the AWS DDoS Response Team (DRT), support experts who apply manual mitigations for more complex and sophisticated DDoS attacks, directly create or update AWS WAF rules, and can recommend improvements to your AWS architectures.It also includes some cost protection against Amazon EC2, Elastic Load Balancing, Amazon CloudFront, and Amazon Route 53 usage spikes that could result from scaling during a DDoS attack
Please take a look at design resilient architecture in aws to mitigate DDOS.
update: If the AWS Shield Advanced team determines that the incident is a valid DDoS attack and that the underlying services scaled to absorb the attack, AWS provides account credit for charges incurred due to the attack. For example, if your legitimate CloudFront data transfer usage during the attack period was 20 GB, but due to the attack you incurred charges for 200 GB of incremental data transfer, AWS provides credit to offset the incremental data transfer charges. AWS automatically applies all credits toward your future monthly bills. Credits are applied towards AWS Shield and cannot be used for payment for other AWS services. Credits are valid for 12 months.
The services covered as per doc are Amazon CloudFront, Elastic Load Balancing, Route 53 or Amazon EC2 . Please check with AWS support, whether your services are covered or not.
There are a couple of options available. First of AWS provides AWS Shield which is a DDoS protection service. The standard subscription is free and covers most frequently occurring network and transport layer DDoS attacks.
On top of that you can consider using AWS WAF - Web Application Firewall which allows you to setup rules for what traffic to allow to your servers.
You can also use API gateway in front of your service and set throttling limits on how much traffic to allow through.
However I would question if you really need this? It sounds like you are worried that you would run up a huge AWS bill if competitors start sending you millions of requests. You can setup billing alerts so when your forecasted bill exceeds a specific threshold you are warned and you can either manually shut down the services that are being bombarded and figure out what the attacks look like, or you can have an automatic response via CloudWatch. I believe that you will find that you will not be under attack and that you should not worry too much on this attack vector at this time.
I have successful secured my backend services using cloud armor and applying white listing or black listing.
Supposing that there is traffic coming from some specific addresses, is there any way to detect them automatically based on the frequency without iterating over the StackDriver logs?
If so is there any way to blacklist them in an automated way?
Cloud Armor does not offer "intelligent" features at the moment. As stated in the Google public docs, it has straight policies for white-listing or blacklisting CIDR ranges. What can be done (not simple, considerable effort required) is to create Stackdriver sink and export the logs for blacklisted IPs. based on the logs captured, there could be cloud function jobs to monitor the logs and then kick off the creation of cloud armor policies to block the offending IPs.But as mentioned, this is not simple. considerable effort required.
I'm new in AWS. For one project we require to purchase server on AWS. I don't know what configuration is required for the server. Our website will be like https://www.justdial.com/ and minimum 1000 users every time will be online on the website. Please, what configuration will be best with minimum pricing. I'm mentioning details below, what we want;
> • 1 - Elastic IP
> • 1 - Load Balancer
> • 2 - Webserver + autoscaling
> • 1 - Database SQL
> • 1 - S3 storage backup
> • CDN
if anything else is missing please guide me.
Go for micro-service architecture, and create labmda for every service. You can use private RDS for security. Using labmda based serverless approach will cost you on the basis of API request per API. Since during night time request reduces to close to zero, for that duration you wont be charged. AWS lambda auto balance load and availability of service to everyone with minimum CPU, memory usage. You won't be needing your load balanced as AWS does it by default.
Based upon your requirements use of a VM won't be a good idea, as most of these, Load-Balancer, Webserver autoscaling, are free for serverless lambda, and using RDS will put your database cost to minimum in place of owning a VM and scaling VM resources.
It really depends on your application. If all you do is return static pages, you might be fine with the smallest instance and CDN like CloudFront. If every request is dynamic and takes massive computations, you need some strong servers.
I suggest you start with some reasonable settings (e.g. t3.medium) and then load-test it to figure out what you really need. There are many tools for that. You basically need something that will generate a lot of requests to your servers and track errors, latency and total response time. If any of those metrics come back insufficient (this also depends on your needs), add more resources. Make sure to leave room for traffic spikes.