I am working with an application deployed on AWS EC2. One head instance, www.me.com, provides login & admin such that a user can spawn an instance of the application. A new AWS EC2 instance is started to run the application, then the user's browser is redirected to that instance (with long URL ending amazonaws.com), and the application is available until the user closes it (then the EC2 instance is stopped).
We now wish to move the application to use SSL. We can get a certificate from some CA, and tie it to *.me.com. The question is, how do we use the same certificate to secure the application instances?
One idea is to use Elastic IP - we reserve N IPs, and tie N sub-domains (foo1.me.com, foo2, ...) to these. Then we start each application instance associated with one of these, and direct the user to the associated sub-domain. Certificate is valid, all is well. I think this works?
Trouble is, the application should scale to 1000's of simultaneous users, but may well spend the majority of time bouncing along around zero users, so we'll pay significant penalty costs for reserving the unused IPs, and besides, we might exceed N and have to deny access.
Simpler, perhaps, would be to provide access to the application routed through the head server, www.me.com, using either of "a.me.com" or "www.me.com/a". The former, I think doesn't work, because it would need the DNS records to be updated to be useful to the user, which cannot happen quickly enough to be offered to a user on the fly. The latter might work, but I don't know web infrastructure well enough to imagine how to engineer it. Even if we were only serving port 80, and in fact we are also providing services on other ports. So we'd need something like:
www.me.com/a:80 <--> foo.amazonaws.com:80
www.me.com/a:8001 <--> foo.amazonaws.com:8001
www.me.com/a:8002 <--> foo.amazonaws.com:8002
...
It seems to me there are two options, either a head server that handles all traffic (under me.com, and thus under the certificate) and somehow hands it off to the application instances, or some method for allowing the users to connect directly to the application instances but in such a way that we can reasonably manage securing these connections using one (or a small number of) certificates.
Can anyone suggest what is the route one way to do this? I'm assuming it's not Route 53, since - again - that's a DNS thing, with DNS lag. Unless I've misunderstood.
Thanks.
Related
I have a k8s cluster that runs my app (gce as an ingress) and I want to restrict access to some endpoints "/test/*" but all other endpoints should be publically available. I don't want to restrict for specific IP's to have some flexibility and ability to access restricted endpoints from any device like phones.
I considered IAP but it restricts access to the full service when I need it only for some endpoints. Hence extra.
I have thought about VPN. But I don't understand how to set this up, or would it even resolve my issues.
I have heard about proxy but seems to me it can't fulfill my requirements (?)
I can't tell that solution should be super extensible or generic because only a few people will use this feature.
I want the solution to be light, flexible, simple, and fulfill my needs at the same time. So if you say that there are solutions but it's complex I would consider restricting access by the IP, but I worry about how the restricted IP's approach is viable in the real life. In a sense would it be too cumbersome to add the IP of my phone every time I change my location and so on?
You can use API Gateway for that. It approximatively meets your needs, it's not so flexible and simple.
But it's fully managed and can scale with your traffic.
For a more convenient solution, you have to use software proxy (or API Gateway), or go to the Bank and use Apigee
I set up OpenVPN.
It was not a tedious process because of the various small obstacles but I encourage you to do the same.
Get a host (machine, cluster, or whatever) with the static IP
Setup an OpenVPN instance. I do docker https://hub.docker.com/r/kylemanna/openvpn/ (follow instructions but update a host -u YOUR_IP)
Ensure that VPN setup works from your local machine
To the routes you need limit IP access to the VPN one. Nginx example
allow x.x.x.x;
deny all;
Make sure that nginx treats IP right. I had an issue that the nginx was having Load Balancer IP as client IP's, so I have to put some as trusted. http://nginx.org/en/docs/http/ngx_http_realip_module.html
Test the setup
I couldn't find anything in the documentation but still writing to make sure I did not miss it. I want all connections from different clients with the same value for a certain request parameter to end up on the same upstream host. With ELB sticky session, you can have the same client connect to the same host but no guarantees across different clients.
This is possible with Envoy proxy, see: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/load_balancers#ring-hash
We already use ELB so if the above is possible with ELB then we can avoid introducing another layer in between with envoy.
UPDATE:
Use-case - in a multi-tenant cloud solution, we want all clients from a given customer account to connect to the same upstream host.
Unfortunately this is not possible to be performed in an ALB.
An application load balancer controls all the logic over which host receives the traffic with features such as ELB sticky sessions and pattern based routing.
If there is no work around then you could look at a Classic Loadbalancer which has support for the application setting the sticky session cookie name and value.
From best practice ideally your application should be stateless, is it possible to look at rearchitecting your app instead of trying work around. Some suggestions I would have are:
Using DynamoDB to store any session based data, moving from a disk based session (if that's what your application does).
Any disk based files that need to persist could be shared between all hosts either using EFS for your Linux based hosts, or FSX on Windows.
Medium/Long term persisting files could be migrated to S3, any assets that rarely change could be stored here and then your application could use S3 rather than disk.
It's important to remember that as I stated above, you should keep your application as stateless as you can. Assume that your EC2 instances could fail, by preparing for this it will make it easier to recover.
I wanted to know if there is an option in Amazon Web Services, two have two EC2 instances running, and me, as a developer, being able to have a direct access to one of my choice when both servers serve under the same domain.
By access, I mean regular access to the website via a web browser (e.g. www.domain.com/some-post/)
I want my site to continue be up and live. I currently have a single EC2 server that servers under www.domain.com. If I add another server via Elastic Load Balancer,I don't have control over which server the load balancer sends me.
I have a Wordpress site which I want to upgrade its theme, plugins and the core files, so I want only me to have access to that server and test it out. I could open a server and test it on a public ip, I did it, and it doesn't work as expected, so I need to run it under the original address to make sure that if it runs OK like that, it will run OK live.
The only way that I thought about doing it is to create an image of the server, create an EC2 instance, use a different domain name, restrict access to the server to my IP address, change in the DB to the new domain name, than after everything works, change the domain back to original and make the Elastic IP point to the new server.
No you can't achieve this behavior with an ELB. This would totally defeat the purpose of an ELB - who's purpose is to evenly distribute traffic amongst the instances associated with it.
By the sounds of it, you're looking for a testing stage that you can use to test out new updates etc without damaging the live site.
You could always set up a DNS name for your domain for your testing stage - eg."alpha.mysite.com".
It's quite common practice to use environment variables for use cases like this. You might have an environment variable set on machines that on prod could be eg: stage=prod and on your testing stage could be stage=test. Then in your code, you can get this environment variable an do something different depending on what stage the code is running on. For example, use the prod/development database.
It might be an idea to start using Code Deploy for pushing your code. This way, you can have deployment hooks set up your environment on each instance - install dependencies, load the code, start the application etc. And then using the environment variables already on the instances being deployed to, your code will do the correct thing.
I suppose you could put the test stage on a different port on your prod machines and that way you could use the same domain, but this would be a really bad idea. I think to get a safe, fault tolerant and scalable solution, you're going to need an additional DNS name. And you most certainly shouldn't use the same ELB. If you want to test load balancing for your test application, you should use an additional ELB.
In fact some people even go the lengths of using different AWS accounts for managing test environments.
You might also be interested in Code Pipeline to help you with this.
If I understand correctly, you run multiple instances behind a single ELB and want to be able to access one of the instances to test upgrades. I assume that, while performance and testing the upgrade, you don't want other users to access that instance.
I can think of a few ways to accomplish this. Here are two practical ones:
1. Remove the instance from the load balancer using the AWS console or CLI. No requests to the ELB will go to this instance.
Access the instance you want to upgrade directly on it's own address. For this, the security group on the instance must be configured to allow HTTP connections from the outsite. You could allow only access from your own IP and the load balancer, for example.
2. Create another ELB for test purposes. Make sure that the instance you're upgrading only responds to the test ELB, not to the production ELB. Two ways to accomplish this: either remove it from the production ELB manually, or make the ELB health check on the instance fail. (in the latter case, you would need different healthchecks for the test and production elb).
My advice: when costs are an issue, go for option one. When the additional costs of an extra ELB is not an issue, go for option 2, manually remove the instance from the production ELB while upgrading, and re-attach it when done and tested.
Update (i realized i didn't answer your question completely): for this to work without changing the domain in your database, you would need to point the machine you're testing from to the right host.
Two options:
1. When going for the direct http connection to the instance, make sure that the instance has an external ip. Put the domain in your hosts file and point it to the ip.
2. When going for an extra test elb, either point the domain in your hosts file to one of the ELB ip's, or run a local dns server that has a record for the domain with a CNAME to the ELB hostname.
Although verifying the correct upgrade of a single production node is a valid use case, in this case you're probably better off creating a separate test environment on a different domain.
This way, you can test all changes/upgrades in isolation.
For best results, you would need to periodically transfer the database from production to the test environment. You could write a database scripts that automatically changes the domain in the database so you can (partially or fully) automate the production-to-test-database-restore process.
I am preparing a system of EC2 workers on AWS that use Firebase as a queue of tasks they should work on.
My app in node.js that reads the queue and works on tasks is done and working and I would like to properly setup a firewall (EC2 Security Group) that allows my machines to connect only to my Firebase.
Each rule of that Security Group contains:
protocol
port range
and destination (IP address with mask, so it supports whole subnets)
My question is - how can I setup this rule for Firebase? I suppose that IP address of my Firebase is dynamic (it resolves to different IPs from different instances). Is there a list of possible addresses or how would you address this issue? Can some kind of proxy be a solution that would not slow down my Firebase drastically?
Since using node to interact with Firebase is outbound traffic, the default security group should work fine (you don't need to allow any inbound traffic).
If you want to lock it down further for whatever reason, it's a bit tricky. As you noticed, there are a bunch of IP addresses serving Firebase. You could get a list of them all with "dig -t A firebaseio.com" and add all of them to your firebase rules. That would work for today, but there could be new servers added next week and you'd be broken. To try to be a bit more general, you could perhaps allow all of 75.126.., but that is probably overly permissive and could still break if new Firebase servers were added in a different data center or something.
FWIW, I wouldn't worry about it. Blocking inbound traffic is generally much more important than outbound (since to generate outbound traffic you have to have already managed to somehow run software on the box)
Questions about load balancers if you have time.
So I've been using AWS for some time now. Super basic instances, using them to do some tasks whenever I needed something done.
I have a task that needs to be load balanced now. It's not a public service though. It's pretty much a giant cron job that I don't want running on the same servers as my website.
I set up an AWS load balancer, but it doesn't do what I expected it to do.
It get's stuck on one server, and doesn't load balance at all. I've read why it does this, and that's all fine and well, but I need it to be a serious round-robin load balancer.
edit:
I've set up the instances on different zones, but no matter how many instances I add to the ELB, it just uses one. If I take that instance down, it switches to a different one, so I know it's working. But I really would like it to always use a different one under every circumstance.
I know there are alternatives. Here's my question(s):
Would a custom php load balancer be an ok option for now?
IE: Have a list of servers, and have php randomly select a ec2 instance. Wouldn't be scalable at all, bu atleast I could set this up in 2 mins and it can work for now.
or
Should I take the time to learn how HAProxy works, and set that up in place of the AWS ELB?
or
Am I doing it wrong, and AWS's ELB does do round-robin. I just have something configured wrong?
edit:
Structure:
1) Web server finds a task to do.
2) If it's too large it sends it off to AWS (to load balancer).
3) Do the job on EC2
4) Report back via curl to an API
5) Rinse and repeat
Everything works great. But because the connection always comes from my server (one IP) it get's sticky'd to a single EC2 machine.
ELB works well for sites whose loads increase gradually. If you are expecting an uncommon and sudden increase on the load, you can ask AWS to pre-warm it for you.
I can tell you I used ELB in different scenarios and it always worked well for me. As you didn't provide too much information about your architecture, I would bet that ELB works for you, and the case that all connections are hitting only one server, I would ask you:
1) Did you check the ELB to see how many instances are behind it?
2) The instances that you have behind the ELB, are all alive?
3) Are you accessing your application through the ELB DNS?
Anyway, I took an excerpt from the excellent article that does a very good comparison between ELB and HAProxy. http://harish11g.blogspot.com.br/2012/11/amazon-elb-vs-haproxy-ec2-analysis.html
ELB provides Round Robin and Session Sticky algorithms based on EC2
instance health status. HAProxy provides variety of algorithms like
Round Robin, Static-RR, Least connection, source, uri, url_param etc.
Hope this helps.
This point comes as a surprise to many users using Amazon ELB. Amazon
ELB behaves little strange when incoming traffic is originated from
Single or Specific IP ranges, it does not efficiently do round robin
and sticks the request. Amazon ELB starts favoring a single EC2 or
EC2’s in Single Availability zones alone in Multi-AZ deployments
during such conditions. For example: If you have application
A(customer company) and Application B, and Application B is deployed
inside AWS infrastructure with ELB front end. All the traffic
generated from Application A(single host) is sent to Application B in
AWS, in this case ELB of Application B will not efficiently Round
Robin the traffic to Web/App EC2 instances deployed under it. This is
because the entire incoming traffic from application A will be from a
Single Firewall/ NAT or Specific IP range servers and ELB will start
unevenly sticking the requests to Single EC2 or EC2’s in Single AZ.
Note: Users encounter this usually during load test, so it is ideal to
load test AWS Infra from multiple distributed agents.
More info at the Point 9 in the following article http://harish11g.blogspot.in/2012/07/aws-elastic-load-balancing-elb-amazon.html
HAProxy is not hard to learn and is tremendously lightweight yet flexible. I actually use HAProxy behind ELB for the best of both worlds -- the hardened, managed, hands-off reliability of ELB facing the Internet and unwrapping SSL, and the flexible configuration of HAProxy to allow me to fine tune how things hit my servers. I've never lost an HAProxy instance yet, but it I do, ELB will just take that one out of rotation... as I have seen happen when the back-end servers have all become inaccessible, which (because of the way it's configured) makes ELB think the HAProxy is unhealthy, but that's by design in my setup.