I'm using lb for prod site. But for internal services (like gitlab, jenkins) I don't want to host a lb. Is there any alternative way to connect to internal services without the use of load balancers? Like could any bastion host do the job?
Having lb for prod and internal services seems to cost around 35 to 45 dollars. I'm trying to reduce the total bill.
I have a nginx ingress controller for production site, wondering if I could do something with it using subdomains for internal services.
Services on a Kubernetes cluster can talk to each other using their ClusterIP, without any use of a load balancer. The IP can be returned using the internal DNS service such as KubeDNS or CoreDNS
For more details
If you need to connect services in different clusters i’d take a look at Kubefed or Submarier for Multi-Cluster deployment.
Which by the way also harness the use of DNS service discovery (this is the term used for in-kubernetes service communication) for service-to-service communication.
Kubefed
Submariner
Related
I use global http(s) load balancer for backend services running on Kubernetes cluster. I didn't find any information on how to limit number of requests in a time window from one IP. There is Cloud Armor, but there also simple IP, region, and header based access can be performed. Could you please share how can I perform IP based rate limitation on global http load balancer on google cloud to provide defence against DoS attacks.
Edit:
The backend service on running on Kubernetes cluster is a symfony server with web interface. I want to use Cloud CDN for the server therefore I had to use gce ingress instead of ingress-nginx. On google cloud, gce ingress creates a global HTTP(s) load balancer and ingress-nginx creates TCP load balancer.
In the nginx-ingress, I could simply use nginx.ingress.kubernetes.io/limit-rps annotation, which helps in limiting flood of http requests. I want to do the similar configuration on my global HTTP(s) load balancer. In the current setting, I observed that flood of http requests are sent to the load balancer which are forwarded to the symfony server and at one point latency of request increases. Which makes the liveness probe fail for the pod.
Cloud Armor is a WAF that you can configure to protect your service against DoS attacks, especially by blocking specific IPs.
Rate limiting isn't to protect your service against DDoS. Indeed, if the attack flood your rate limiting service, your valid IPs and the bad IPs won't be served, because your service is flooded: it's a denial of service
Rate limiting helps to preserve resource for legit users. But some can try to overcome some constraint by using in a different (wrong/bad) manner your APIs.
For example, you have a paid API to export all the customer. You have a free API to request 1 customer. A user can say "Hey I don't want to pay, I will request in a loop the single customer API to create my daily report like that!". It's a misuse of the single customer API and you can protect it against this misuse with rate limiting
You can use Cloud Endpoint and ESP (Extensible Service Proxy). I wrote an article with an ESP deployed on Cloud Run, but you can deploy it on K8S also.
You can also use API gateway which is the managed service of ESP, that will be soon plugable on HTTPS load balancer (to use it in addition to WAF protection).
We are using Spring Cloud Netflix Eureka for Service Registration. We will be deploying all microservices in GCP (Google Cloud).
Environment
We have Eureka Servers running as a cluster.
Eureka Server registers themselves as client to its peer in application.properties
eureka.client.service-url.default-zone=http://xx.xx.xx.xxx:8762/eureka
Client microservices register/enroll themselves by
providing Eureka Server IPs in application.properties
eureka.client.service-url.default-zone=http://xx.xx.xx.xxx:8761:/eureka,http://xx.xx.xx.xxx:8762:/eureka
Since IP Address and hostnames are dynamic in cloud, can we configure Eureka Servers in cluster without using ipaddress/hostname.
Please provide a sample confiugration to use in Google Cloud.
gcloud maintains internal DNS resolver for subnets (if you are using default OS images).
So you can use host names to resolve IP addresses. Like prod-redis-2.c.project-<id>.internal.
You may probably need to configure links between subnets to avoid making IP addresses public.
I have not used GCP but have implemented and deployed spring cloud on PCF (which, on a higher level, is pretty much same as GCP).
You cannot make defaultZone completely dynamic. Why? Because these propeties are picked up during the application startup.
There needs to be something (some service or database) in your architecture that tells your services the dynamic hostnames/IP-addresses of other services. That is Eureka server in your case. All services needs to know the address (hostname/IP-address) of Eureka service. Now if Eureka server's hostname is dynamic, then how will your services know about the new hostname of Eureka server when that hostname changes?
You'll have to update the address of Eureka server manually only. What, at max, you can do is externalize defaultZone to a centralized configuration server (or something similar). That way you'll have to update the new address at one place only.
I am setting up a Rancher environment.
The Rancher server is behind a classic ELB (since ALBs are not recommended per Rancher guidelines).
I also want to make available Prometheus and Grafana services.
These are offered via Rancher catalogue and will run as container services, being exposed on Rancher host ports 3000 and 9090.
Since Rancher server (per their recommendations) requires ELB, I wanted to explore the options on how to make available the two services above using the most minimal possible setup.
If the server is available on say rancher.mydomain.com, ideally I would like to have the other two on grafana.mydomain.com and prometheus.mydomain.com.
Can I at least combine the later two behind an ALB?
If so, how do I map them?
Do I place <my_rancher_host_public_IP>:3000 and <my_rancher_host_public_IP>:9090 behind an ALB?
You could do this a couple (maybe more) ways:
use an external dns updater like the route 53 infra catalog item. That will automatically map dns directly to the public ip of the host that houses the services. Modify the dns template so it prepends the service name to the domain.
register your targets and map the ports, then set a dns entry to the ALB.
The first way will allow for dns to update in case the service shifts across hosts in your environment. You could leverage the second way and force containers to specific hosts.
I would like to ask you if you have an microservice architecture (based on Spring Boot) involving Amazon Elastic Container Service (ECS) with Application Load Balancer(ALB), service discovery is performed automatically by the platform, or do you need a special mechanism (such as Eureka or Consul)?
From the documentation (ECS and ALB) is not clear you have this feature provided.
I have talked this with the Amazon support team and they respond the following:
"...using Service Discovery on AWS ECS[..] just with ALBs.
So, there could be three options here:
1) Using ALB/ELB as service endpoints (Target groups for ALBs, separate ELBs if using ELBs)
2) Using Route53 and DNS for Service Discovery
3) Using a 3rd Party product like Consul.io in combination with Nginx.
Let me speak about each of these options.
Using ALBs/ELBs
For this option the idea is to use the ELBs or ALB Target groups in front of each service.
We define an Amazon CloudWatch Events filter which listens to all ECS service creation messages from AWS CloudTrail and triggers an Amazon Lambda function.
This function identifies which Elastic Load Balancing load balancer (or an ALB Target group) is used by the new service and inserts a DNS resource record (CNAME) pointing to it, using Amazon Route 53.
The Lambda function also handles service deletion to make sure that the DNS records reflect the current state of applications running in your cluster.
The down side here is that it can incur higher costs if you are using ELBs - as you need an ELB for each service. And it might not be the simplest solution out there.
If you wish to read more on this you can do so here[1]
Using Route53
This approach involves the use of Route53 and running a simple agent[2] on your ECS container instances.
As your containers stop/start the agent will update the Route53 DNS records. It creates a SRV record. Likewise it will delete said records once the container is stopped.
Another part of this method is a Lambda function that performs health checks on ECS container instances - and removes them from R53 in case of a failure.
You can read up more on this method, on our blog post here[3].
Using a 3rd Party tool like Consul.io Using tools like Consul.io on ECS, will work - but is not supported by AWS. So you are free to use it, but we - unfortunately - do not offer support for it.
So, in conclusion - there are a few ways of implementing service discovery on AWS ECS - the two ways I showed here that use AWS resources, and of course the way of using 3rd party applications.
"
you dont have an out-of-the-box solution in AWS, although it is possible with some effort as described in https://aws.amazon.com/es/blogs/compute/service-discovery-an-amazon-ecs-reference-architecture/
You may also install Zuul + Ribbon + Eureka or Nginx + Consul and use ALB to distribute traffic among Zuul or Nginx
We are using Eureka with AWS ECS service that can scale docker containers.
In ECS if you leave out the host port, or specify it as being '0', in your task definition, then the port will be chosen automatically and reported back to the service. After the task is running, describing it should show what port(s) it bound to.
How does Eureka can resolve what port to use if we have several EC2 instance. For example Service A from EC2-A try to call Service B from EC2-B. So Eureka can resolve hostname , but cannot identify exposed port
Hi #Aleksandr Filichkin,
I don't think Application Load Balancer and service registry does the same.
The main difference traffic flows over the (application) load balancer whereas the service registry just gives you a healthy endpoint that your client directly can address (so the network traffic does not flow over the service registry).
Cheap is a very relative term, maybe it's cheap for some, maybe it's an unnecessary overhead for others.
The issue was resolved
https://github.com/Netflix/eureka/issues/937
Currently ECS agent knows about running port.
But I don't recommend to use Eureka with ECS, because Application Load Balancer does the same. It works as service registry and discovery. You don't need to run addition service(Eureka), ALB is cheap.
There is another solution.
You can create an application loadbalancer and a target group, in which the docker containers can be launched.
Every docker container has set their hostname to the hostname of the loadbalancer. If you need a pretty url, then you can utilize Route53 for DNS-Routing.
It looks like this:
Service Discovery with Loadbalancer-Hostname
Request Flow
If you have two containers of the same task on different hosts, both will communicate the same loadbalancer hostname to eureka.
With this solution you can use eureka with docker on AWS ECS without loosing the advantages and flexibility of dynamic port mapping.