I have a Flask API running on a single EC2 instance. I need to add SSL (https) to my app and the docs seem to indicate that the best way is to use ELB.
The problem is when I set up the EC2 behind the ELB and with a Route 53 pointing to the load balancer, I get latency issues like never before. API calls that should take 150ms, take 31s. If I ping the EC2 directly, there is no issue, but pinging the Route53/ELB takes too long.
I've looked at past responses such as these:
The reason for the delay is because you have the ELB setup for
multi-az without any application instances in the other 2 AZ's
configured. Without instances in those AZ's requests will tend to fail
because the ELb still returns IP addresses for those AZ's even if
there are no active application instances. Please disable the other
AZ's for now and continue your tests.
So I deleted the subnets and AZs that are not related to my EC2 instance, but now I have:
503 errors: Backend server is at capacity all over the place.
About to pull my hair out. All I am trying to do it setup SSL for my app...
For anyone else struggling with this, using the Application Load Balancer instead of the Classic Load Balancer has resolved this latency issue.
Seems like some connectivity issue. Follow this step to drill down the issue. As its happening from load balancer only. So first do the NSLOOKUP like nslookup name. You will get minimum two IP.
Try to individual curl LB IP's n check the response time. You may use below command or perhaps any of your fav tool.
curl -w "dns_resolution: %{time_namelookup}, \
tcp_established: %{time_connect}, \
ssl_handshake_done: %{time_appconnect}, \
TTFB: %{time_starttransfer}\n" \
-o /dev/null -s "http://<ELB IP>/<path>"
This way you be figure out whether is the issue. It be due to routing issue on one specific AZ subnet, or some other issue.
Related
I am running an app in AWS Beanstalk, I use jenkins to do automatic deploys, manage crons, ecc, jenkins connects to the EC2 behind Beanstalk using the public ip.
The problem arises when the instance scales, since the IP of the EC2 will be different, I have to manually update Jenkins every time.
One of the simplest options would be to open the port 22 in the loadbalancer, but since I am using the recommended application loadbalancer, it only allows me to open the port 80/443. I was wondering if there is a way to create a dns record in route 53, that will automatically point to the right IP every time it scales?
I would like to avoid changing load balancer, because there are at least 20 environments that will need to be reconfigured.
I tried to look but no-one seems to have this issue, so either I have the wrong architecture, or it is too easy to fix.
I'm running through this tutorial to create a deployment pipeline with my custom .net-based docker image.
But when I start a deployment, it's stuck on install phase, so I have to stop it manually:
After that I get a couple of running tasks with different task definitions (note :1 and :4, 'cause I've tried to run deployment 4 times by now):
They also change their state RUNNING->PROVISIONING->PENDING all the time. And the list of stopped tasks grows:
Q:
So, how to hunt down the issue with CodeDeploy? Why It's running forever?
UPDATE:
It is connected to health checks.
UPDATE:
I'm getting this:
(service dataapi-dev-service, taskSet ecs-svc/9223370487815385540) (port 80) is unhealthy in target-group dataapi-dev-tg1 due to (reason Health checks failed with these codes: [404]).
Don't quite understand, why is it failing for newly created container, 'cause the original one passes health-check.
While the ECS task is running, ELB (Elastic Load Balancer) will constantly do healthchecking the container as you config in the target group to check if the container is still responding.
From your debug message, the container (api) responded the healthcheck path with 404.
I suggest you config the healhcheck path in target group dataapi-dev-tg1.
For those who are still hitting this issue: in my case the ECS cluster had no outbound connectivity.
Possible solutions to this problem:
make security groups you use with your VPC allow outbound traffic
make sure that the route table you use with VPC has subnet associations with subnets you use with your load balancer (examine route tables)
I have able to figure it out because I enabled CloudWatch during ECS cluster creation and got CannotPullContainerError. For more information on solving this problem look into Cannot Pull Container Image Error.
Make sure your Internet Gateway is attached to your Subnets through the Route Table (Routes), if your Load Balancer is internet facing.
The error is due to health check which detected an unhealthy target.
Make sure to check your configuration in Target group settings.
I am deploying a laravel installation in AWS, everything runs perfectly when I allow it to recieve all inbound traffic (EC2>Network&Security>Security Groups>Edit inbound rules.), if I turn off inbound traffic and limit it to an IP it doesnt load the webpage it gives me this error:
PDO Exception SQLSTATE[HY000] [2002] Connection timed out
However for security reasons I dont want this setup like this, I dont want anyone being able to even try to reach my webapp. Everything is being hosted in AWS, I dont have any external entities, its running in RDS and EC2. I added en elastic IP address and whitelisted it, but that didnt work either. I followed every step in this tutorial : http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/php-laravel-tutorial.html#php-laravel-tutorial-generate
Environmental variables are working as well as dependencies, well.. pretty much everything unless I restrict inbound traffic as I mentioned.
How do I whitelist AWS own instance then to make this work with better security?
Thank you!
I think part of this answer is what you may be looking for.
You should enable inbound access from the EC2 security group associated with your EC2 instance, instead of the EC2 IP address.
More than just adding an elastic IP address to your AWS instance you need to do two more things.
Assign the elastic IP to your AWS instance ( yes is not the same as just adding it to the instance, you must specify )
White list the internal IP that it generates once you link it to your app.
?????
Profit
This is my setup on Amazon Web Services
EC2 instances
Autoscaling + loadbalancer
Route 53 with A-record pointing to loadbalancer
.fi-domain
This setup normally works fine. However, under heavy traffic mydomain.fi will occasionally not respond. The odd thing is, however, that connecting directly to the loadbalancer's public DNS works fine.
In my mind the problem lies in one of the following:
The LB connects to unhealthy/not initialized instance
Problem with Route53 (misconfiguration?)
(Problem with the .fi-domain)
I guess the most likely problem is individual EC2 instances not working. I currently do not use any health checks to check if the instances are responding. Could this help? Any tips on best practice with this and Route 53 + LB?
Thanks a lot!
I would say the problem is precisely because you are not using any health checks. How can the ELB know if your ec2 instances are healthy and responding, if you have not setup healthchecks?
The fact that the problem only occurs during period of high activity only further points to the likelihood that this is your problem.
If you've gone to the trouble of setting up an ELB, you need to also setup the health checks so it get do what it is supposed to do - avoid sending traffic to node that can't handle it.
Questions about load balancers if you have time.
So I've been using AWS for some time now. Super basic instances, using them to do some tasks whenever I needed something done.
I have a task that needs to be load balanced now. It's not a public service though. It's pretty much a giant cron job that I don't want running on the same servers as my website.
I set up an AWS load balancer, but it doesn't do what I expected it to do.
It get's stuck on one server, and doesn't load balance at all. I've read why it does this, and that's all fine and well, but I need it to be a serious round-robin load balancer.
edit:
I've set up the instances on different zones, but no matter how many instances I add to the ELB, it just uses one. If I take that instance down, it switches to a different one, so I know it's working. But I really would like it to always use a different one under every circumstance.
I know there are alternatives. Here's my question(s):
Would a custom php load balancer be an ok option for now?
IE: Have a list of servers, and have php randomly select a ec2 instance. Wouldn't be scalable at all, bu atleast I could set this up in 2 mins and it can work for now.
or
Should I take the time to learn how HAProxy works, and set that up in place of the AWS ELB?
or
Am I doing it wrong, and AWS's ELB does do round-robin. I just have something configured wrong?
edit:
Structure:
1) Web server finds a task to do.
2) If it's too large it sends it off to AWS (to load balancer).
3) Do the job on EC2
4) Report back via curl to an API
5) Rinse and repeat
Everything works great. But because the connection always comes from my server (one IP) it get's sticky'd to a single EC2 machine.
ELB works well for sites whose loads increase gradually. If you are expecting an uncommon and sudden increase on the load, you can ask AWS to pre-warm it for you.
I can tell you I used ELB in different scenarios and it always worked well for me. As you didn't provide too much information about your architecture, I would bet that ELB works for you, and the case that all connections are hitting only one server, I would ask you:
1) Did you check the ELB to see how many instances are behind it?
2) The instances that you have behind the ELB, are all alive?
3) Are you accessing your application through the ELB DNS?
Anyway, I took an excerpt from the excellent article that does a very good comparison between ELB and HAProxy. http://harish11g.blogspot.com.br/2012/11/amazon-elb-vs-haproxy-ec2-analysis.html
ELB provides Round Robin and Session Sticky algorithms based on EC2
instance health status. HAProxy provides variety of algorithms like
Round Robin, Static-RR, Least connection, source, uri, url_param etc.
Hope this helps.
This point comes as a surprise to many users using Amazon ELB. Amazon
ELB behaves little strange when incoming traffic is originated from
Single or Specific IP ranges, it does not efficiently do round robin
and sticks the request. Amazon ELB starts favoring a single EC2 or
EC2’s in Single Availability zones alone in Multi-AZ deployments
during such conditions. For example: If you have application
A(customer company) and Application B, and Application B is deployed
inside AWS infrastructure with ELB front end. All the traffic
generated from Application A(single host) is sent to Application B in
AWS, in this case ELB of Application B will not efficiently Round
Robin the traffic to Web/App EC2 instances deployed under it. This is
because the entire incoming traffic from application A will be from a
Single Firewall/ NAT or Specific IP range servers and ELB will start
unevenly sticking the requests to Single EC2 or EC2’s in Single AZ.
Note: Users encounter this usually during load test, so it is ideal to
load test AWS Infra from multiple distributed agents.
More info at the Point 9 in the following article http://harish11g.blogspot.in/2012/07/aws-elastic-load-balancing-elb-amazon.html
HAProxy is not hard to learn and is tremendously lightweight yet flexible. I actually use HAProxy behind ELB for the best of both worlds -- the hardened, managed, hands-off reliability of ELB facing the Internet and unwrapping SSL, and the flexible configuration of HAProxy to allow me to fine tune how things hit my servers. I've never lost an HAProxy instance yet, but it I do, ELB will just take that one out of rotation... as I have seen happen when the back-end servers have all become inaccessible, which (because of the way it's configured) makes ELB think the HAProxy is unhealthy, but that's by design in my setup.