We are using GCP external HTTPS load balancer, architecture is shown in the diagram below. The primary use of LB is redirecting users to static error site (hosted on Cloud Storage bucket) in case CE instance is down, Traefik crashes on CE, Docker crashes on CE, etc.
We have 4 backend services defined on load balancer:
static-error-page backend bucket
blog-backend-service, gallery-backend-service and shop-backend-service zonal network endpoint groups
Then, we defined host and path rules so that:
https://blog.company.com -> blog-backend-service
https://gallery.company.com -> gallery-backend-service
https://shop.company.com -> shop-backend-service
All unmatched (default) -> static-error-page
Each zonal network endpoint group (blog-backend-service, gallery-backend-service and shop-backend-service) has just 1 endpoint defined: 192.168.171:443 (192.168.171.3 is internal IP of CE instance).
However, since my websites are served behind reverse proxy (Traefik), specifying IP:port combination in network endpoint is useless because they all have the same IP:port. I would like to specify URL instead of IP:port in network endpoint (that way network endpoint would also show correct health status if website is down, it always reports healthy now, even if application is down).
Is it possible to specify URL instead of IP:port in network endpoint? If not, what are my alternatives?
Instead of using treafik, you can use Google API Gateway which was meant to do that job, while you can still use the load balancer behind it.
Related
I use global http(s) load balancer for backend services running on Kubernetes cluster. I didn't find any information on how to limit number of requests in a time window from one IP. There is Cloud Armor, but there also simple IP, region, and header based access can be performed. Could you please share how can I perform IP based rate limitation on global http load balancer on google cloud to provide defence against DoS attacks.
Edit:
The backend service on running on Kubernetes cluster is a symfony server with web interface. I want to use Cloud CDN for the server therefore I had to use gce ingress instead of ingress-nginx. On google cloud, gce ingress creates a global HTTP(s) load balancer and ingress-nginx creates TCP load balancer.
In the nginx-ingress, I could simply use nginx.ingress.kubernetes.io/limit-rps annotation, which helps in limiting flood of http requests. I want to do the similar configuration on my global HTTP(s) load balancer. In the current setting, I observed that flood of http requests are sent to the load balancer which are forwarded to the symfony server and at one point latency of request increases. Which makes the liveness probe fail for the pod.
Cloud Armor is a WAF that you can configure to protect your service against DoS attacks, especially by blocking specific IPs.
Rate limiting isn't to protect your service against DDoS. Indeed, if the attack flood your rate limiting service, your valid IPs and the bad IPs won't be served, because your service is flooded: it's a denial of service
Rate limiting helps to preserve resource for legit users. But some can try to overcome some constraint by using in a different (wrong/bad) manner your APIs.
For example, you have a paid API to export all the customer. You have a free API to request 1 customer. A user can say "Hey I don't want to pay, I will request in a loop the single customer API to create my daily report like that!". It's a misuse of the single customer API and you can protect it against this misuse with rate limiting
You can use Cloud Endpoint and ESP (Extensible Service Proxy). I wrote an article with an ESP deployed on Cloud Run, but you can deploy it on K8S also.
You can also use API gateway which is the managed service of ESP, that will be soon plugable on HTTPS load balancer (to use it in addition to WAF protection).
I have setup an External HTTP(S) load balancer with the following:
2 Serverless NEGs, each pointing at a different Cloud Run service in their respective region
1 Backend Service, using the 2 NEGs as 2 Backends
1 Host and path rule that sends everything to the Backend Service
1 HTTPS Frontend pointing at the Host and path rule
At this point, I notice that the traffic is routed to the Cloud Run service closest to the region of the client making the request.
I would like to change that to route 100% of the traffic to one Cloud Run service on day 1, 50% on each service on day 2, and on day 3, route 100% of the traffic to the other Cloud Run service.
It's unclear if an External HTTP(S) load balancer can help with that. And if it can, it's unclear if this should be done in the Backend Service or in the Host and Path rule.
Google Cloud load balancer does not support weighted/percent-based load balancing for the external HTTP(S) LB. This is listed at https://cloud.google.com/load-balancing/docs/features#load_balancing_methods.
Maybe I need to create 2 Backend Services, each pointing at one NEG?
Yes, this is how you would do it if external HTTPS GCLB supported it. You need to create separate backendServices for each serverless NEG and list weightedBackendServices in the route rule of the urlMap object. You can find an example here but I believe it only works for internal load balancer (ILB) currently per the link above.
AFAIK, External HTTPS load balancing can only route to the closest location but not dispatch the traffic according to weight.
In addition, your solution requires to deploy in 2 different regions, because you can't 2 backends in the same region in the same backend service.
The easiest solution for now is to use Cloud Run traffic splitting feature. Route all the traffic to the same service, and then, let the Cloud Run load balancer dispatching the requests.
I have a running Web server on Google Cloud. It's a Debian VM serving a few sites with low-ish traffic, but I don't like Cloudflare. So, Cloud CDN it is.
I created a load balancer with static IP.
I do all the items from the guides I've found. But when it comes time to Add origin to Cloud CDN, no load balancer is available because it's "unhealthy", as seen by rolling over the yellow triangle in the LB status page: "1 backend service is unhealthy".
At this point, the only option is to choose Create a Load Balancer.
I've created several load balancers with different attributes, thinking that might be it, but no luck. They all get the "1 backend service is unhealthy" tag, and thus are unavailable.
---Edit below---
During LB creation, I don't see anywhere that causes the LB to know about the VM, except in cert issue (see below). Nowhere does it ask for any field that would point to the VM.
I created another LB just now, and here are those settings. It finishes, then it's marked unhealthy.
Type
HTTP(S) Load Balancing
Internet facing or internal only?
From Internet to my VMs
(my VM is not listed in backend services, so I create one... is this the problem?)
Create backend service
Backend type: Instanced group
Port numbers: 80,443
Enable Cloud CDN: checked
Health check: create new: https, check /
Simple host and path rule: checked
New Frontend IP and port
Protocol: HTTPS
IP: v4, static reserved and issued
Port: 443
Certificate: Create New: Create Google-managed certificate, mydomain.com and www.mydomain.com
Load balancer's unhealthy state could mean that your LB's healthcheck probe is unable to reach your backend service(Your Debian VM in this case).
If your backend service looks good now, I think there is a problem with your firewall configuration.
Check your firewall rules whether it allows healthcheck probe's IP address range or not.
Refer to the docoment below to get more detailed information.
Required firewall rule
Need some serious help here, thanks a lot in advance !
I need to deploy a scalable 3 tier web application on AWS and I am having some doubts/trouble understanding the best practice to design the architecture.
NOTE: As per my understanding, all the backend requests are requested through the browser, after the Frontend server serves html/css/js to the user.
Let me show you what I have come up with till now :
Assuming the above 'note':
Cons (as per my understanding):
All the backend routes will be exposed to the outside world.
Even though backend servers are in private subnet, now that they're being accessed via external load balancer, the endpoints API could be accessed from the users.
How will we route a request from a Load balancer to another Load balancer. Because what I have seen is that you could only route a request to an EC2 instance added in the target group.
To overcome the cons as I think in the above approach, I came up with this architecture instead:
Pros (as per my understanding):
The backend routes are safe (in a way) because we have a way of internally connecting from the frontend to the backend servers(if required).
Cons:
If the request is made from the browser, the endpoints are again exposed.
Solution that I found online:
REAL BIG DOUBT IN THIS LAST ONE
This breaks all the logic of my understanding that : All the requests are made by the browser from the user to the backend because in this the requests to the backend are being routed FROM the frontend servers.
QUESTIONS
What if the backend request (say login) is made by the user from the browser?
How will this work out in such case?
seems like you have done some good work here.
Let me start by making things easy for you:
Users only interact with the Load Balancer: If you want to keep it simple and not break off your frontend asset serving to an external service like CloudFront, which you should if you are starting out, you will be hosting the application only via EC2 instances (application origin, or simply orgin). Your requests would look something like this:
Users <--> ALB <--> EC2
Notice how users never interact with EC2 instances directly, its always via Application Load Balancer (ALB).
If I can oversimply thing, this is how HTTP operates, a request is made to a resource at an IP and the response is sent back from the same resource or IP. So as in your diagram, a request will not be responded back by EC2 but rather be relayed via the ALB.
You don't need NAT gateway: NAT gateway are there to make it possible for resources in provate subnet access the internet. In this case, unless you want your application to access the internet, you don't need NAT gateway. Many large scale applications are actually locked down in part by not keeping this resource at all.
You are still protecting the origin: Given that only the ALB can be accessed over the internet and everything else internal you can structure things here in any way that you want to. you could have few internal microservices that can be used internally without ever being exposed to end users. Note that here request never leaves the VPN.
You can read more about this and build a sample application via the official docs here or access AWS tutorials here.
To me, #3 is the correct solution because it does not expose /api to end users (since you mention "I DO NOT want the users to directly access the /api"). In #1, I don't think you could limit access to /api to only the front-end servers, since security groups work on the whole load balancer, not per-target.
Also, being an Internet-facing load balancer, any requests from the front-end servers to the load balancer in #1 will be referencing the load balancer via public IP addresses. This will cause a 1c/GB charge to go "out of" the VPC and then back in again.
Only #3 correctly refers to back-end resources via private IP addresses. The internal load balancer will be referenced via private IP addresses.
I have a Node-Express website running on a microservices based architecture. I deployed the microservices on Amazon ECS cluster with one EC2 instance. The microservices sit behind an Application Load Balancer that routes external traffic correctly to the services. This system is working as expected except for one problem: I need to make a POST request from one service to the other. I am trying to use axios for this but I don't know what url to post to in axios. When testing locally, I just used axios.post('http://localhost:3000/service2',...) inside service 1 but how should I do it here?
So There are various ways.
1. Use Application Load Balancer behind the service
In this method, you put your micro services behind the load balancer(s) and to send request, you give load balancer URL. You can have path based routing for same load balancer or you can use multiple load balancers.
2. Use Service Discovery
In this method, you let your requester discover it. Now Service discovery can be done in various way like using ALB or Route 53 or ECS or Key Value Store or Configuration Management or Third Party Software such as Consul