I have a cluster running in DigitalOcean, and need to connect a RDS hosted Database. What I would normally do is just to whitelist the IP addresses of the machines that would be accessing the database in the AWS Security Groups, and then have clear access to the Database.
The thing is that DigitalOcean Kubernetes nodes get recycled every now and then, and the IP changes. I can manually change the IP to allow connection from every node in my cluster, but this doesn't seem like a very solid solution.
How would I go about fixing the IP of the cluster, or maybe making some sort of gateway with a fixed IP that every outgoing connection from the cluster would go through that? I have been studying but I'm completely ignorant in networking, so anything that could help me point to the right direction would be great.
I've seen there are Forward Proxies (Gateways from what I've read), but I couldn't find much information on that, much of my research ends up in API Gateway (like Kong or something), which I understand is the exact opposite of what I need.
Any help?
Related
our company just moved to a new office and therefore also got new network equipment. Es it turns out, our new firewall does not allow pushing routes over VPN that it first has to look up ip addresses for.
As we all know, amazon aws does not allow static ip addresses for its application load balancer.
So our idea was to simply put a network load balancer in front of the application load balancer (there is a pretty hacky way described by aws itself (https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/) that seemed to work fine (even if I don't really like the approach with the lambda script registering and deregistering targets)
So here is our problem: as it turns out, the application load balancer only gets to see the network load balancers ip address. This prevents us to use security groups for ip whitelisting which we do quite heavily. On top of that some of our applications (Nginx/PHP based) also do ip address verification and the alb used to pass the clients ip address as an x-forwarded-for header. Now our application only sees the one from the nlb.
We know of the possibility to use the global accelerator but that is a heavy investment as we don't really need what the GA is trying to solve.
So how did you guys solve this problem ?
Thankful for any help :)
Greetings
You could get the list of AWS IP addresses for the region your ALB is located, and allow for them in your firewall. They do publish the list and you can filter through it https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html
I haven't done this myself and I'm unsure if the addresses for ALB are included under the EC2 category of you would take the whole of AMAZON service "to be safe".
Can you expand on this? "We know of the possibility to use the global accelerator but that is a heavy investment as we don't really need what the GA is trying to solve."
GA should give you better, more consistent performance, especially if your office is far away from the AWS Region where the ALB is running
I want to use Kubernetes on some clouds (maybe Amazon, Google, etc). Should I disallow my EC2 machines from accessing the external network? My guess is as follows, and I wonder whether it is correct or wrong?
I should disallow EC2 from accessing the external network. Otherwise, hackers can attack my machines more easily. (true?)
How to do it: I should use a dedicated load balancer (maybe Ingress) with the external IP that my domain name is bound to. The load balancer will then talk with my actual application (which has no external IP and can only access internal network). (true?)
Sorry I am new to Ops, and thanks for any help!
Allowing or disallowing your EC2 instances from accessing external networks, ie keeping the rule that allows all outgoing traffic in your security group won't be of much use keeping hackers out, that's what the incoming traffic rules are for. It will, however, prevent unwanted traffic from going out after the hacker has reached your instance and has been able to install whatever malicious software on it, and then it would try to initiate outgoing communication.
That outgoing traffic rule is usually kept to allow things like getting software installs and updates, but it won't affect how your instances respond to incoming requests (legitimate or not).
It is a good idea to have a load balancer in front of your instances and have it be the only allowed point of entry to your services. It's a good pattern to follow, and your instances will not need to have an external IP address.
Having a bastion host is a good idea as well, and use it to manage the instances themselves. And I would also recommend Systems Manager's Session Manager for this task.
I setup a Fargate cluster on AWS. My cluster has the following services:
server-A (port 3000)
server-B (port 4000)
Each service is in the same VPC and have the same security group (any ports, any source, any destination). The VPC is isolated from internet.
Now, I want server-A to send a http query to server-B. I would assume that, as in Docker swarm, there is a private DNS that maps the service name to its private IP, and it would be as simple as sending the query to: http://server-B:4000. However, server-A gets a timeout, which means it can't reach server-B.
I've read in the documentation that I can put the 2 containers in the same service, each container listening on a different port, so that, thanks to the loopback interface, from server-A, I could query http://127.0.0.1:4000 and server-B will respond, and vice-versa.
However, I want to be able to scale server-A and server-B independently, so I think it makes sense to keep each server independant from each other by having 2 services.
I've read that, for 2 tasks to talk to each other, I need to setup a load balancer. Coming from the world of Docker Swarm, it was so easy to query the services by their service name, and behind the scene, the request was forwarded to one of the containers in that service. But it doesn't seem to work like that on AWS Fargate.
Questions:
how can server-A talk to server-B?
As service sometimes redeploy, their private IP changes, so it makes no sense to query by IP, querying by hostname seems the most natural way
Do I need to setup any kind of internal DNS?
Thanks for your help, I am really lost on doing this simple setup.
After searching, I found out it was due to the fact that I was not enabling "Service Discovery" during the service creation, so no private DNS was created. Here is some additional documentation which explains exactly the steps:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-discovery.html
Here's what's bothering me. Is there a better way than sending emails to devs that the ip address for their dev server has changed after the instance is stopped and started?
I was thinking of a single small instance that has an elastic ip which the devs can log in using terminal, and ssh again to the internal ip address of the dev server. Is that effective?
Does it mean that the devs need to be informed of the change every time?
It's unclear exactly what you are saying "there's a new public dns for the server"? -thanks for the comment, that's clearer what you mean! It's the aws domain name in the format "ec2-54-222-213-143.eu-west-1.compute.amazonaws.com" you are referring to
You are asking how can these name/address changes be managed?
Generally speaking for fixing these kinds of problems there are a couple of things to be aware of
Firstly, if it is the public ip address that is changing instead of an ephemeral public ip address use an elastic ip. This will stay the same and can be transferred from an old instance to a new instance. Please read http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html about the differences between "Elastic IP" and normal public IP addresses on AWS
Secondly, if you are concerned about maintenance of the dns records that map the ip addresses to the domain names then it is possible to automate the updates to aws route53. I have used the aws cli command "route53 change-resource-record-sets" for this and also CloudFormation
Automating events to occur on instance start up does take a little research of the available APIs and hooks for example see this answer with a simple use of cloud-init Using cloud-init user data
Questions about load balancers if you have time.
So I've been using AWS for some time now. Super basic instances, using them to do some tasks whenever I needed something done.
I have a task that needs to be load balanced now. It's not a public service though. It's pretty much a giant cron job that I don't want running on the same servers as my website.
I set up an AWS load balancer, but it doesn't do what I expected it to do.
It get's stuck on one server, and doesn't load balance at all. I've read why it does this, and that's all fine and well, but I need it to be a serious round-robin load balancer.
edit:
I've set up the instances on different zones, but no matter how many instances I add to the ELB, it just uses one. If I take that instance down, it switches to a different one, so I know it's working. But I really would like it to always use a different one under every circumstance.
I know there are alternatives. Here's my question(s):
Would a custom php load balancer be an ok option for now?
IE: Have a list of servers, and have php randomly select a ec2 instance. Wouldn't be scalable at all, bu atleast I could set this up in 2 mins and it can work for now.
or
Should I take the time to learn how HAProxy works, and set that up in place of the AWS ELB?
or
Am I doing it wrong, and AWS's ELB does do round-robin. I just have something configured wrong?
edit:
Structure:
1) Web server finds a task to do.
2) If it's too large it sends it off to AWS (to load balancer).
3) Do the job on EC2
4) Report back via curl to an API
5) Rinse and repeat
Everything works great. But because the connection always comes from my server (one IP) it get's sticky'd to a single EC2 machine.
ELB works well for sites whose loads increase gradually. If you are expecting an uncommon and sudden increase on the load, you can ask AWS to pre-warm it for you.
I can tell you I used ELB in different scenarios and it always worked well for me. As you didn't provide too much information about your architecture, I would bet that ELB works for you, and the case that all connections are hitting only one server, I would ask you:
1) Did you check the ELB to see how many instances are behind it?
2) The instances that you have behind the ELB, are all alive?
3) Are you accessing your application through the ELB DNS?
Anyway, I took an excerpt from the excellent article that does a very good comparison between ELB and HAProxy. http://harish11g.blogspot.com.br/2012/11/amazon-elb-vs-haproxy-ec2-analysis.html
ELB provides Round Robin and Session Sticky algorithms based on EC2
instance health status. HAProxy provides variety of algorithms like
Round Robin, Static-RR, Least connection, source, uri, url_param etc.
Hope this helps.
This point comes as a surprise to many users using Amazon ELB. Amazon
ELB behaves little strange when incoming traffic is originated from
Single or Specific IP ranges, it does not efficiently do round robin
and sticks the request. Amazon ELB starts favoring a single EC2 or
EC2’s in Single Availability zones alone in Multi-AZ deployments
during such conditions. For example: If you have application
A(customer company) and Application B, and Application B is deployed
inside AWS infrastructure with ELB front end. All the traffic
generated from Application A(single host) is sent to Application B in
AWS, in this case ELB of Application B will not efficiently Round
Robin the traffic to Web/App EC2 instances deployed under it. This is
because the entire incoming traffic from application A will be from a
Single Firewall/ NAT or Specific IP range servers and ELB will start
unevenly sticking the requests to Single EC2 or EC2’s in Single AZ.
Note: Users encounter this usually during load test, so it is ideal to
load test AWS Infra from multiple distributed agents.
More info at the Point 9 in the following article http://harish11g.blogspot.in/2012/07/aws-elastic-load-balancing-elb-amazon.html
HAProxy is not hard to learn and is tremendously lightweight yet flexible. I actually use HAProxy behind ELB for the best of both worlds -- the hardened, managed, hands-off reliability of ELB facing the Internet and unwrapping SSL, and the flexible configuration of HAProxy to allow me to fine tune how things hit my servers. I've never lost an HAProxy instance yet, but it I do, ELB will just take that one out of rotation... as I have seen happen when the back-end servers have all become inaccessible, which (because of the way it's configured) makes ELB think the HAProxy is unhealthy, but that's by design in my setup.