I have an ec2 linux server with a spring boot application and nginx reverse proxy. It has been running for months.
Today I tried accessing the server and it was not reachable, nor via ssh, nor via http.
I went to cloudWatch and saw these metrics :
I don't know how to interpret this : EBSReadBytes raised to 8 GB for an hour.
To gain access to the server I had to restart the instance. After that I went to nginx access logs :
Last java logs :
I don't see anything unusual in nginx logs. (The censored lines are from legit users)
For the java app, I usually have these kind of logs on my development computer when I let it go to sleep and comeback, but never had these on my linux server.
Do you have an idea why this happened ?
I am now pretty sure it was a resource leak due to not closing a file input stream.
Related
I have a running instance on the Linux 2 AMI.
I have a default VPC and network interface.
Security groups taken care of, even opened all traffic and still got nothing.
There is an Internet Gateway
Routes are open on the VPC
The server is running
nginx is running
Once the instance is initiated and installed, all of this is ready
I can reach the http website the first 2-3 minutes, then it is unreacheable.
No idea why, everything else still running, can still ssh into the server, but http port 80 not running.
I opened everything from iptables, still nothing.
If I reboot the server, I get a minute where I can reach the server via http, but then a minute later its the same again.
I can reach http if I use $ wget http://localhost
So I think it is probably something from the EC2 control panel, not the instance itself.
I tried on new instances too.
Anyone has an idea?
The reason behind this weird behavior was that AWS abuse team had blocked some of my ports, had to upgrade to the developer plan to be able to know this, contacting them at the moment
I have a problem with my Amazon EC2 instance (that did not happened when I was using DigitalOcean).
I've several EC2 instances that are managed by me. My personal EC2 has about 5 Wordpress sites running on a t2.micro instance and the traffic is not high so it is working well in load speed.
Also I have another 2 instances for one of my clients, one t2.micro (running only one Wordpress site) and a t3a.micro (running 4 Wordpress sites). The issue is with all 3 instances (mine and all the 2 of my client).
I have a CloudWatch alarm to notify me by email when Error 504 happen. Since I get the alarm, the website becomes unavailable (Cloudflare shows me Error 504), but I can get into SSH or Webmin. I do service nginx status and all seems to be fine, same to service php7.2-fpm. I do pkill nginx && pkill php* and then service nginx start && service php7.2-fpm start correctly but when I try to enter to the site, the Error 504 is still there.
To test, I decided to install and configure Apache with and without PHP-FPM enabled, same problem. Instance going well and websites running fast but after X amount of hours, it becomes unaccessible via web and the only solution is rebooting...
What's the only thing that solve the issue? Well, rebooting the instance.... After it boots, the websites are available again. Please note that I moved from DigitalOcean to AWS because it is more useful but I can't understand why the problem is happening here and not there since I've a similar instance configured very similar...
In all of the instances I've a setup with:
OS: Ubuntu 18.04
Types: Two t2.micro and one t3a.micro
ELB: Enabled
Security Groups: only allow ports 80, 443 from all the sources.
Database: In a RDS, not on the same instance.
I can provide the logs of everything that you probably can ask but I review all the Nginx and PHP-fpm logs and I can't see any anomalies. Also with syslog and kern.log, but I can provide if it can helps.
Hope you can give me a hand. Thanks for your advice!
EDIT:
I already found the origin of the issue. The problem wasn't in the EC2, all my headache was because I have the RDS set with only one Security Group attached to allow access from my IP to remote management of the databases and the public IPs of the EC2 that runs Wordpress, but I figured that I also need to whitelist the private IPs of those EC2s... Really noob mistake but that was the solution.
I've just finished setting up my site on a free Amazon Web Services EC2 Ubuntu server.
I'm not very knowledgeable in deployment, and I'm not 100% clear on what Nginx or gunicorn even is, but I'm following a tutorial to launch a Django project.
While doing things the same exact way, having no errors, I have noticed that sometimes I will go to my site and get 'refused to connect' or 'taking too long to respond.'
One of my previous projects had no issue, one of them never loaded the page, and the last one I did gave me this problem which was cured by rebooting the server.
I've rebooted the server several times as well as deactivated and reactivated the venv (as a classmate suggested) but it isn't working. I noticed that last night my terminal just kept taking forever to load and the Amazon web services site was just being slow as well.
Is this just Amazon's fault? Is there anything I can do?
You are spinning up your server. You are responsible to manage it.
There are a couple of things you need to check. The problem could be service may not be listening on a different port (check on IP as well), inbound and outbound security groups might not be configured right.
Amazon is not responsible for anything you do with their resources. It is a company to provide resources to simplify your business.
You can read AWS SLA here,
https://aws.amazon.com/s3/sla/
I am running Bitnami WordPress on AWS server website working since two days but suddenly it stop showing anything and connection timeout is showing. The instance EC2 is running perfectly fine, and I have also seen IP logs, and nothing suspicious has come up.
Based on the above comments I guess the issue is with the internal web server
Make sure that the web server is running perfectly fine. And I do not mean just checking the EC2 instance state, because it is possible that the EC2 instance is running but the web server is down, causing the issue
My Amazon EC2 small instance stopped responding, I looked at the AWS console and CPU use had gone through the roof. I tried rebooting instance but it didn't respond. So I stopped it and started it again (twice).
Now says the CPU usage is fine (was triggering an alarm when breaching 90%) but still can't login via SSH and Apache is not working (my sites are down).
Anyone give me any idea how I can sort this out? I'm out of my depth a bit as unfamiliar with the ins and outs of EC2.
EDIT: console log http://pastebin.com/JWFeG7NU shows Apache, SSH, etc starting up fine but I can't access via SSH and no response to pinging website hosted on server.
If you have stop/started your instance and you were not using an elastic IP address, your instance IP has changed.
If you were using an elastic IP address, it would have become disassociated.
If you do have applications that are causing you to exceed the allocated CPU, other applications such as ssh, may become slow to respond or not respond at all within the timeout.