Environment status becomes severe when second environment is deployed - django

I uploaded sample django project to AWS using elasticbeanstalk.
I deployed two environments accoding to a tutorial here https://colintoh.com/blog/how-to-deploy-application-to-aws-elastic-beanstalk.
When I deployed(create using eb cli) the second environment, the first environment helth changes to sever showing errors on elasticbeanstak console
100.0 % of the requests are erroring with HTTP 4xx. Insufficient request rate (12.0 requests/min) to determine application health.
ELB processes are not healthy on all instances.
ELB health is failing or not available for all instances.
However when I access both pages they seem to be working correctly since they both show django's debug message for first page
The install worked successfully! Congratulations!
You are seeing this page because DEBUG=True is in your settings file and you have not configured any URLs.
Can I just ignore the error or I need to do something to fix the error(and how to fix it)?


Normal behaviour on applications with zero traffic.
A little more advanced but you could add a healtcheck path to your loadbalancer, it will generate 'some' traffic.
Also in the console there is an option (environment -> configuration -> monitoring) to ignore 4XX error's.


By default, Elastic Beanstalk checks the health of your environment by sending a request to / path and expects it return a 200 code.
Maybe this route does not exist on your project.
If you need to, you can configure the path where EB will send a request to check the health. In the configuration panel of your environment, go to Load Balancer and edit the default process to change the Health check path.

Related

Problem routing traffic to AWS elastic beanstalk environment

In my AWS environment I have 2 hosted zones:
aaa.nl
bbb.nl
I have an elastic beanstalk (NodeJS) application running. It works fine. I want this EB application to be available via the domain name:
my-app.bbb.nl
I followed this approach:
https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-beanstalk-environment.html#routing-to-beanstalk-environment-create-alias-procedure
Configuring a new record is easy. I can select my EB application and the record is created successfully. And i see it listed in the record list of hosted zone bbb.nl (A Record)
Now I open up my browser and navigate to:
my-app.bbb.nl
To my great surprise, what I get to see in the browser is a different application that I have running at https://my-other-app.aaa.nl, but the url is https://app.bbb.nl (my-other-app is listed as an A record in hosted zone aaa.nl)
I did get some certificate warnings in my browser as well (in the browser I see that it has the ssl certificate of aaa.nl coming along),
I do have something else running correctly at https://bbb.nl
I have no idea how this is possible and how to debug this. Anyone any clues?

Use browser development tools to trace the request to my-app.bbb.nl
Look for redirection in response.
Check if my-other-app.aaa.nl isn't hardcoded into app itself.

AWS EBS application timing out when changed to a single instance environment

I have a web application running on Elastic Beanstalk in load balanced environment however when I changed the configuration to a "single instance" environment the application returns a 408 Request Timeout with every https browser request to the server (custom domain).
The environment health in my AWS console shows everything is running okay so I am baffled by what could be causing the problem. When I change the configuration back to 'load balanced' everything works fine again.

When I change the configuration back to 'load balanced' everything works fine again.
Since you are using HTTPS with custom domain, when you switch to a single instance, the HTTPS functionality is lost. To make HTTPS work on a single instance, you need to obtained new SSL certificate (AWS ACM can't be used), and deploy it on your instance though re-configured Nginx:
How to Setup SSL(HTTPS) on Elastic Beanstalk Single Instance Environment

Use AWS Beanstalk Hooks to do an HTTP GET request

Is there a way to do an HTTP GET call during AWS Beanstalk deployment and make it roll back to previous version in case of an error response, even though the application has been successfully deployed.

You can use the ELB health check to customise which of your application endpoints is checked for application health, see docs.
There are slightly different settings depending on which kind of load balancer you're using, eg classic or application.

How can I configure an automatic timeout for an Elastic Load Balancer?

Does anyone know of a way to make Amazon's Elastic Load Balancers timeout if an HTTP response has not been received from upstream in a set timeframe?
Occasionally Amazon's Elastic Beanstalk will fail an update and any requests to the specified resource (running Nginx + Node if tht's any use) will hang any request pages whilst the resource attempts to load.
I'd like to keep the request timeout under 2s, and if the upstream server has no response by then, to automatically fail over to a default 503 response.
Is this possible with ELB?
Cheers

You can Configure Health Check Settings for Elastic Load Balancing to achieve this:
Elastic Load Balancing routinely checks the health of each registered Amazon EC2 instance based on the configurations that you specify. If Elastic Load Balancing finds an unhealthy instance, it stops sending traffic to the instance and reroutes traffic to healthy instances. For more information on configuring health check, see Health Check.
For example, you simply need to specify an appropriate Ping Path for the HTTP health check, a Response Timeout of 2 seconds and an UnhealthyThreshold of 1 to approximate your specification.
See my answer to What does the Amazon ELB automatic health check do and what does it expect? for more details on how the ELB health check system work.

TLDR - Set your timeout in Nginx.
Let's see if we can walkthrough the issues.
Problem:
The client should be presented with something quickly. It's okay if it's a 500 page. However, the ELB currently waits 60 seconds until giving up (https://forums.aws.amazon.com/thread.jspa?messageID=382182) which means it takes a minute before the user is shown anything.
Solutions:
Change the timeout of the ELB
Looks like AWS support will help increase the timeout (https://forums.aws.amazon.com/thread.jspa?messageID=382182) so I imagine that you'll be able to ask for the reverse. Thus, we can see that it's not user/api tunable and requires you to interact with support. This takes a bit of lead time and more importantly, seems like an odd dial to tune when future developers working on this project will be surprised by such a short timeout.
Change the timeout of the nginx server
This seems like the right level of change. You can use proxy_read_timeout (http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout) to do what you're looking for. Tune it to something small (and in particular, you can set it for a particular location if you would like).
Change the way the request happens.
It may be beneficial to change how your client code works. You could imagine shipping a really simple html/js page that 1. pings to see if the job is done and 2. keeps the user updated on the progress. This takes a bit more work then just throwing the 500 page.

Recently, AWS added a way to configure timeouts for ELB. See this blog post:
http://aws.amazon.com/blogs/aws/elb-idle-timeout-control/

Why does Elastic Load Balancing report 'Out of Service'?

I am trying to set up Elastic Load Balancing (ELB) in AWS to split the requests between multiple instances. I have created several images of my webserver based on the same AMI, and I am able to ssh into each individually and access the site via each distinct public DNS.
I have added each of my instances to the load balancer, but they all come back with the Status: Out of Service because they failed the health check. I'm mostly confused because I can access each instance from its public DNS, but I get a timeout whenever I visit the load balancer DNS name.
I've been trying to read through all the docs and googling it, but I'm stuck. Any pointers or links in the right direction would be greatly appreciated.

I contacted AWS support about this same issue. Apparently their system doesn't know how to handle cases were all of the instances behind the ELB are stopped for an extended amount of time. AWS support can manually refresh the statuses, if you need them up immediately.
The suggested fix it to de-register the ec2 instances from the ELB instead of just stopping them and re-register them when you start again.

Health check is (by default) made by accessing index.html on each instance incorporated in load balancer. If you don't have index.html in document root of instance - default health check will fail. You can set custom protocol, port and path for health check when creating elastic load balancer.

Finally I got this working. The issue was with the Amazon Security Groups, because I've restricted the access to port 80 to few machines on my development area and the load balancer could not access the apache server on the instance. Once the load balancer gained access to my instance, it gets In Service.
I checked it with tail -f /var/log/apache2/access.log in my instance, to verify if the load balancer was trying to access my server, and to see the answer the server is giving to the load balancer.
Hope this helps.

If your web server is running fine, then it means the health check goes on a url that doesn't return 200.
A trick that works for me : go on the instance, type curl localhost:80/pathofyourhealthcheckurl
After you can adapt your health check url to always have a 200 response.

In my case, the rules on security groups assigned to the instance and the load balancer were not allowing traffic to pass between the two. This caused the health check to fail.

I to faced same issue , i changed Ping Protocol from https to ssl .. it worked !
Go to Health Check --> click on Edit Health Check -- > change Ping protocol from HTTPS to SSL
Ping Target SSL:443
Timeout 5 seconds
Interval 30 seconds
Unhealthy Threshold 5
Healthy Threshold 10

For anyone else that sees this thread as this isn't listed:
Check that the health check is checking the port that the responding server is listening on.
E.g. node.js running on port 3000 -> Point healthcheck to port 3000;
Not port 80 or 443. Those are what your ALB will be using.
I spent a morning on this. Yes.

I would like to provide you a general way to solve this problem. When you have set up you web server like apache or nginx, try to read the access log file to see what happened. In my occasion, it report 401 error because I have add the basic auth in nginx. Of course, just like #ivankoni remind, it may because of the document you check is not exist.

I was working on the AWS Tutorial on hosting a web app and ran into this problem. Step 7b states the following:
"Set Ping Path to /. This sends queries to your default page, whether
it is named index.html or something else."
They could have put the forward slash in quotations like this "/". Make sure you have that in your health checks and not this "/." .

Adding this because I've spent hours trying to figure it out...
If you configured your health check endpoint but it still says Out of Service, it might be because your server is redirecting the request (i.e. returning a 301 or 302 response).
For example, if your endpoint is supposed to be /app/health/ but you only enter /app/health (no trailing slash) into the health check endpoint field on your ELB, you will not get a 200 response, so the health check will fail.

I had a similar issue. The problem appears to have been caused due to my using a HTTP health check and also using .htaccess to password protect the site.

I got the same error, in my case had to copy the particular html file from s3 bucket to "/var/www/html" location. The same html referenced in load balancer path.
The issue resolved after copying html file.

I had this issue too, and it was due to both my inbound and outbound rule for the Load Balancer's Security Group only allowing HTTP traffic on port 80. I needed to add another rule for HTTPS traffic on port 443.

I was also facing that same issue,
where ELB (Classic-Load-Balancer) try to request /index.html not / (root) while health check.
If it unable to find /index.html resource it says 'OutOfService'. Be Sure index.html should be available.