How to determine root cause of AWS Elastic Beanstalk Shutdown Errors - django

Question

I have a Django app hosted on AWS Elastic Beanstalk.
Users upload documents to the site. Sometimes, users upload documents and the server completely shuts down. The server instantly 500s, goes offline for about 4 minutes and then then magically the app is back up and running.
Obviously, something is happening to the app where it gets overwhelmed.
The only thing I get from Elastic Beanstalk is this message:
Environment health has transitioned from Ok to Severe. 100.0 % of the requests are failing with HTTP 5xx. ELB processes are not healthy on all instances. ELB health is failing or not available for all instances.
Then about 4 minutes later:
Environment health has transitioned from Severe to Ok.
I have 1 t2.medium EC2 instance. I've set it up as Load Balancing, but use Min 1 Max 1, so I don't take advantage of the load balancing features.
Here's a screenshot of my health tab:
My app shut off on 7/10 as can be seen in picture 1. My CPU spiked at this time, but I can't imagine 20% CPU was enough to overwhelm my server.
How can I determine what might be causing these short 500 errors? Is there somewhere else I can look to discover the source of this? I don't see anything helpful in my access_log or error_log. I don't know where to start looking.

Answer 1

I was having similar problems with Elastic Beanstalk without using load balancer. So when I faced that problem, my application was simply crashing and I needed to rebuild the environment from scratch. Further search revealed that the problem was sometimes the EC2 memory was exceeding, which was causing the elastic beanstalk to shutdown. The solution was adding a swap area (I preferred 2048MB of swap) and prevent these sudden memory exceeding.
Here is how to add swap area to the elastic beanstalk instance:
.ebextensions/swap-area.sh:
#!/usr/bin/env bash
SWAPFILE=/var/swapfile
SWAP_MEGABYTES=2048
if [ -f $SWAPFILE ]; then
echo "$SWAPFILE found, ignoring swap setup..."
exit;
fi
/bin/dd if=/dev/zero of=$SWAPFILE bs=1M count=$SWAP_MEGABYTES
/bin/chmod 600 $SWAPFILE
/sbin/mkswap $SWAPFILE
/sbin/swapon $SWAPFILE
.ebextensions/00-swap-area.config:
container_commands:
00_swap_area:
command: "bash .ebextensions/swap-area.sh"
Then after the deployment, you can check the swap area by using commands such as top etc. in your EC2.