I have an ubuntu 18.04 based EC2 instance using an Elastic IP Address. I am able to SSH into the instance without any problems.
apt is executing some unattended updates on the instance. If I reboot the system after the updates, I am no longer able to SSH into the system. I am getting the error ssh: connect to host XXX port 22: Connection refused
Few points:
Even after the updates, I am able to SSH before the reboot
Method of restart does not make a difference. sudo shutdown -r now and EC2 dashboard have the same result.
There are no problems with sshd_config. I've detached the volume and attached it to a new working instance. sshd -t did not report any problems either
I am able to do sudo systemctl restart ssh.service after the updates but before the system restart.
I've tried with and without Elastic IP. Same result
From the system logs, I see that SSH is trying to start, but failing for some reason
I want to find out why the ssh daemon is not starting. Any pointers?
Update:
System Logs
Client Logs
No changes in the security groups before and after reboot
EC2 > Network & Security > Security Groups > Edit inblound rules > SSH 0.0.0.0/0
Step 1: EC2 > Instances > Actions > Image and templates > Create image
Step 2: Launch a new instance using the AMI image.
I missed the error Failed to start Create Static Device Nodes in /dev. in system logs. The solution given at https://askubuntu.com/questions/1301750/ubuntu-16-04-failed-to-start-create-static-device-nodes-in-dev helped solve my problem
Related
I've been trying to connect an ec2 machine from aws to my localhost (WSL on windows) docker swarmer cluster, but i keep getting displayerd : Error response from daemon: Timeout was reached before node joined. The attempt to join the swarm will continue in the background. Use the "docker info" command to see the current swarm status of your node." and the ec2 is not being added as a node (even if later I try to add it again, it says that it is already part of a cluster, on my localhost it does not appear added).
What I've been tried:
Open the doors 2377, 7946 and 4789 (required by docker) on my wsl and ec2.
approved all traffic to all ports on my ec2 firewall.
desable my windows firewall (Tried to init a windows cluster to add ec2, but did not worked too.
Aditional information:
to open the doors on my wsl/ec2 I mainly used ufw and telnet.
I was able to connect my windows docker to my wsl cluster.
I'm being able to ping my ec2 ipv4 adress from mylocalhost, but not my localhost ip from ec2.
Any suggestions and solutions are welcome, i'm seriously HOURS in this, any progress will make me happy
Systems: I'm using ubuntu 18-04 on wsl and ecs, and windows 11
Please find below ports and protocols open on the security group my ec2 is using
ssh -i "LightsailDefaultKey-ap-south-1.pem" bitnami#[ip-of-lightsail-instance]
ssh: connect to host 6[ip-of-lightsail-instance] port 22: Connection timed out
UPSTREAM_NOT_FOUND
An error occurred and we were unable to connect or stay connected to your instance. If this instance has just started up, try again in a minute or two.
UPSTREAM_NOT_FOUND [519]
PUTTY says
Connection Timeout
Create a snapshot and add script
sudo ufw disable
sudo iptables -F
sudo mv /etc/hosts.deny /etc/hosts.deny_backup
sudo touch /etc/hosts.deny
echo "Port 22" >> /etc/ssh/sshd_config
systemctl restart sshd
sudo systemctl enable sshd
sudo systemctl restart sshd
Wait 10-15minutes. Done! Issue fixed :-)
Ran into the same problem. Managed to log in after rebooting from the browser. My problem started after some upgrades and updates and heavy installations that took up most of my 512MB memory. The solution going forward is to create a swapfile to improve the performance of the system.
I struggled with this 519 Upstream Error for several days in Lightsail as well. I could not connect via SSH-in-Browser, nor via SSH in my Mac Terminal. It simply timed out.
However, I found a solution that works -
In short, you can:
Create a snapshot of the "broken server"
Export to Amazon EC2 (think of EC2 as a more manipulatable version of Lightsail)
Create a new volume from that snapshot in EC2, and mount it on a new machine image
There is a great video here that I followed step by step:
https://www.youtube.com/watch?v=EN2oXVuOTSo
After following those steps, I was able to SSH in to the new volume, and recover all my data in the /mnt directory. You may need to change some permissions with chown to access the data.
Were you able to make your instance working, or it was just data retrieval?
For only data and files, EC2 is not required. You can use AWS cli to create disksnapshot, then create disk, and attach to any instance, mount and then access files.
My SSH connection was done after changing SSH port, just a reboot and use the new port I could connect again.
If this still didn't work, you could resort to the official Adding a Lightsail instance to Systems Manager while launching section, I followed to create a new instance from the snapshot, the new instance is reachable by SSH.
About 6 months ago I created an AWS EC2 instance to mess around with on the free tier. After months of having no issues remoting into my AWS EC2 server, I've recently been unable to access it via SSH. I am using the following command:
ssh -i my-key-pair.pem ec2-user#ec2-**-**-***-***.us-****-*.compute.amazonaws.com
...and after a minute or two, am getting this response
ssh: connect to host ec2-**-**-***-***.us-****-*.compute.amazonaws.com port 22: Operation timed out
What's strange is that
1) I can read and write to my RDS database just fine
2) I can ping into the server
3) My port 22 is open
4) The instance is running and healthy
5) In the Inbound section of the security group of the EC2 server it allows for all traffic and SSH from any location via port 22.
6) I'm using the same key-pair as always
I went through this documentation (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) and can confirm that the VPC, subnet, network ACL and route tables all line up (I haven't changed anything with those since the SSH stopped working). Any insight would be extremely helpful!
Sometimes the instance fails, you can check the screen of it via AWS
console.
Run another instance in the same security group and try to
connect to it and then from there to your original one - to verify if
ssh is still open (even if you do not have the ssh key, the error
will not be 'timeout')
You can create a snapshot of your instance and
attach it as another volume in a new one and you can investigate
logs, maybe something went wrong.
You can restart the instance, if
for example i ran out of memory it will most likely work after the
reboot (hopefully for a long enough time for you to investigate).
You can contact AWS support.
I'm trying to use docker-machine with my docker instance hosted on amazon aws.
I run the following command:
$ sudo docker-machine create --driver generic --generic-ip-address={EC2 IP} --generic-ssh-key ~/.ssh/id_rsa dockeraws
Running pre-create checks...
Creating machine...
(dockering) Importing SSH key...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Error creating machine: Error detecting OS: Too many retries waiting for SSH to be available. Last error: Maximum number of retries (60) exceeded
But It stucks on 'Waiting for SSH to be available...' and I don't know why.
I've also opened the ports '22' and '2376' but it's still not working.
For my instance I'm using the template stated on the docker page here -> https://docs.docker.com/docker-for-aws/
Try adding your machine ip address to allowed host to the security group used by your ec2 instance. This solved the issue for me.
Generate a ssh key (if you didn't):
ssh-keygen
Then install your public key in the server using ssh-copy-id:
ssh-copy-id user#remote-server
Where user is your remote user and remote-server your server IP/URL.
Anybody please help me
Im unable to connect my server after run this command sudo ufw allow 'Nginx Full'.
In aws is there any option to undo this changes or anything else
Thanks in advance
Stop the running EC2 instance
Detach its /dev/sda1 volume (let's call it volume A)
Start the new t1.micro EC2 instance, create it on the same subnet, otherwise you will have to terminate the instance and create it again.
Attach volume A to the new micro instance, as /dev/xvdf
SSH to the new micro instance and mount volume A to /mnt/tmp
Disable UFW by setting ENABLED=no in /mnt/tmp/etc/ufw/ufw.conf
Exit
Terminate micro instance
Detach volume A from it
Attach volume A back to the main instance as /dev/sda1 Start the main instance
Login as before
Source
If you have server backup, try restoring to that backup.
If not, try looking at AWS Troubleshooting Guide.
Please post your error or logs upon connecting. Can't help much without logs.
After struggling for 2 days I found few easy alternatives, here are those:
Use AWS session manager to connect with out ssh or key (yt)
Use EC2 serial console
Update the user instance details (link)