SSH into EC2 instance while it is running a script - amazon-web-services

I have an ec2 instance that I am using to populate a database. I have a python script that I run by entering:
nohup python populate_db.py &. This works and the database starts getting populated. However, if I leave the ssh session and then try to rejoin as this is running I get the following output when adding the vvv flag:
OpenSSH_7.4p1, OpenSSL 1.0.2k 26 Jan 2017
debug1: Reading configuration data /home/me/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "ec2-xx-xxx-xxx-x.compute-1.amazonaws.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ec2-xx-xxx-xxx-x.compute-1.amazonaws.com
[xx.xxx.xxx.x] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file .ssh/ec2key.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file .ssh/ec2key.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
ssh_exchange_identification: read: Connection reset by peer
Additionally my cloudwatch logs seem to stop recording, and the CPU usage after spiking to a maximum of 80% drops down to around 6% usage, so it feels like something might be crashing.
Is there a way to figure out what is going wrong/fix this?

It turns out that my script had a memory leak in it. I didn't have memory monitoring over on EC2, but running it locally the memory in use climbed to 2.5 GB within a few minutes (much more than the t2.micro can handle).

Related

Failed to connect to EC2 even security group is correct

I created an EC2 instance, when ssh to it, it says " debug1: Connection established." , then hangs there and time out. I can also telnet to it with port 22. That mean the server is reachable, and security group is correct, right? but why ssh times out?
ssh -i "devops.pem" ec2-user#ec2-3-91-100-189.compute-1.amazonaws.com -vvvv
OpenSSH_8.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/test/.ssh/config
debug1: /Users/test/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/test/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/test/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/test/.ssh/master-ec2-user#ec2-3-91-100-189.compute-1.amazonaws.com:22" does not exist
debug1: Connecting to ec2-3-91-100-189.compute-1.amazonaws.com port 22.
debug1: Connection established.
debug1: identity file devops.pem type -1
debug1: identity file devops.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
It's blocked by firewall. When fixed the firewall, the connection is good.

ssh into emr instance failing with ssh_exchange_identification error

I'm attempting to ssh into an emr instance. The ip and key information are correct. Ideas as to what the problem might be?
session trace:
> ssh -vv -i pemfile.pem hadoop#xx.xx.xx.xx
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file pemfile.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file pemfile.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
ssh_exchange_identification: Connection closed by remote host
> ls -al pemfile.pem
-rw-r--r-- 1 user staff 1692 Aug 24 15:09 pemfile.pem
This can be caused by a number of issues, steps to test are as follows.
Check /etc/hosts.deny and /etc/hosts.allow
Many have reported configuring these to values properly has helped. However, I seen this issue even when hosts.* files were not a factor.
root#host # grep sshd /etc/hosts.allow
sshd: ALL
Missing Dependencies
This one typically happens after a glibc or openssl upgrade. Many distros can install updates to glibc or openssl libs and not require a restart to sshd.
On any distro you can recognize it when after an update, then use lsof to see where sshd has open files. Some will be pointed to DEL, because those libs were deleted on update.
~ # lsof -n | grep ssh | grep DEL
When an SSH connection comes in the sshd daemon forks and attempts to attach (ld) these lib files, and fails resulting in this error.
Corrupted Fingerprint / Keys
Some how one or the other of the fingerprints or keys has become corrupted (did you manually edit one of these files?). Remove the server-side fingerprint in the clients ~/.ssh/known_hosts and try again. When you re-connect you will be prompted to accept the host identity again.
If you are able to access the machine another way you may want to back-out and re-create the server-side ~/.ssh/authorized_keys.
Along the same lines as this issue, if the files /etc/ssh/key are removed and sshd is not restarted then this error will show up too. Check for the key files in the sshd configuration directory.
Heavy Server Load
Have also seen this happen when server was under heavy load from for example, brute force attack. Increase the amount of connections sshd can run.
root#host # grep MaxStartups /etc/ssh/sshd_config
# Old Style
MaxStartups 12
# New Style
MaxStartups 10:20:30
Reference: http://edoceo.com/notabene/ssh-exchange-identification

ec2 instance ssh access blocked

I was trying just for fun to configure an openvpn server on an ec2 instance. After the last reboot I wasn't able to connect anymore to my instance; the following is the client log:
macbrein:~ renato$ ssh ec2-user#52.62.20.211 -i .ssh/ec2-openvpn.pem -vvv
OpenSSH_6.9p1, LibreSSL 2.1.7
debug1: Reading configuration data /Users/renato/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: /etc/ssh/ssh_config line 102: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 52.62.20.211 [52.62.20.211] port 22.
debug1: connect to address 52.62.20.211 port 22: Connection refused
ssh: connect to host 52.62.20.211 port 22: Connection refused
this is the last log of the instance (retrieved in the aws control panel):
Starting openvpn: [10.975933] tun: Universal TUN/TAP device driver, 1.6 [10.979593] tun: (C) 1999-2004 Max Krasnyansky <maxk#qualcomm.com>
Enter Private Key Password:
apparently openvpn is stuck because it's prompting for the private key's password and I think this is blocking everything else, might this be the problem?
This is not a production machine, i can simply terminate it and start a new one from scratch, but I'm wondering if there is a more appropriate way to deal with this kind of problems!
thank you!

AWS SSH connection error: Permission denied (publickey)

Im trying to connect to my EC2 instance with SSH and Iḿ getting crazy. I have read this post and tried all user combinations:
AWS ssh access 'Permission denied (publickey)' issue
Its still not working for me. Any idea what am I missing?
roberto#ubuntu:~/keys$ ssh -v -i ec2-key-pair.pem ec2-user#ec2-54-72-242-0.eu-west-1.compute.amazonaws.com
OpenSSH_6.6, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to ec2-54-72-242-0.eu-west-1.compute.amazonaws.com [54.72.242.0] port 22.
debug1: Connection established.
debug1: identity file ec2-key-pair.pem type -1
debug1: identity file ec2-key-pair.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6p1 Ubuntu-2ubuntu1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA e4:06:ee:a5:a5:d2:97:5f:0f:b7:06:5e:f2:b3:da:26
debug1: Host 'ec2-54-72-242-0.eu-west-1.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /home/roberto/.ssh/known_hosts:3
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: ec2-key-pair.pem
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
UPDATE:
According to #aldanux suggestions:
roberto#ubuntu:~/keys$ ssh-keygen -R 54.72.242.0
# Host 54.72.242.0 found: line 4 type ECDSA
/home/roberto/.ssh/known_hosts updated.
Original contents retained as /home/roberto/.ssh/known_hosts.old
roberto#ubuntu:~/keys$ ssh -i ec2-key-pair.pem ec2-user#ec2-54-72-242-0.eu-west-1.compute.amazonaws.com
Warning: Permanently added the ECDSA host key for IP address '54.72.242.0' to the list of known hosts.
Permission denied (publickey).
You are probably logging in as the wrong user. If it's a Ubuntu instance the command would be:
ssh -v -i ec2-key-pair.pem ubuntu#ec2-54-72-242-0.eu-west-1.compute.amazonaws.com
While not specific to AWS, this unhelpful error message
debug1: key_parse_private2: missing begin marker
will occur under a handful of obscure scenarios, such as when the ownership (or the permissions) on the SSH user's home directory are incorrect on the remote machine.
The best way to troubleshoot this and similar obscure messages is to examine the authorization log on the remote machine, provided you have access, as it will usually pinpoint the problem. On Debian and Ubuntu systems, this is most easily accomplished with tail (use sudo as appropriate):
tail -f -n 80 /var/log/auth.log
In my particular case, I found
Authentication refused: bad ownership or modes for directory /var/www
Perfectly accurate and concise: the owner:group was set to daemon:daemon when it should have been www-data:www-data (this was on a Ubuntu machine that must have had some other web-server installed in the past).
I had a similar issue, "key_parse_private2: missing begin marker" while using username 'ec2-user' but it got fixed when I changed to ubuntu as the user.
Try this steps:
ssh-keygen -R 54.72.242.0
sudo chmod 600 ec2-key-pair.pem
and then:
ssh -i ec2-key-pair.pem ec2-user#ec2-54-72-242-0.eu-west-1.compute.amazonaws.com
Another thing to check is PermitRootLogin and AllowUsers in /etc/ssh/sshd_config.
This debug1: key_parse_private2: missing begin marker appears even after successful key authorization if your user access restricted.
Yes, indeed quite misleading message. In my case I used wrong key for instance.
We had need to removed key pair and created new one, except that our instance kept using old one(because you can't do it that easy).
The error message was the same so it's worth to check key name in your aws panel of instance match the key pair that you use in key paris.
Logging in as "admin" worked for me. Based on your instance type the login user changes. ec2-user or ubuntu or in my case admin.
ssh -v -i ./my_key_file.pem admin#ec2-11-222-333-44.compute-1.amazonaws.com
Also ensure the permission for the pem file is 600
chmod 600 ./my_key_file.pem
One easy way to get this error is a corrupt .pem file.
For example, if the last line is missing, you get "missing begin marker".
Make sure the .pem ends with:
-----END RSA PRIVATE KEY-----
Many problems may cause the connectivity issue:
Please review the following settings:
AWS security group settings and check the ssh port 22 policy
Check the firewall setting you are using in your lan connection
Generate the ssh-keygen on your local machine and add to aws linux server for future safety.
Regenerate the new ssh key in the panel.
Check your ipblacklist on mxtoolbox if you are using firewall on the aws linux server.
Please try above all if possible to overcome the error. Let me know if its working or not.

AWS ssh access 'Permission denied (publickey)' issue [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
How to connect to a AWS instance through ssh?
I have:
Signed up at AWS;
Created a public key and a certificate at AWS website and saved them to disk;
Went to my console and created environment variables:
$ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
$ export EC2_CERT=/home/default/aws/cert-EBAINCRNWHDSCWWIHSOKON2YWGJZ5LSQ.pem
$ export EC2_PRIVATE_KEY=/home/default/aws/pk-EBAINCRNWHDSCWWIHSOKON2YWGJZ5LSQ.pem
Told AWS API to use this keypair and saved the keypair to file:
$ ec2-add-keypair ec2-keypair > ec2-keypair.pem
Started an AWS Ubuntu 9 instance using this keypair:
$ ec2-run-instances ami-ed46a784 -k ec2-keypair
Attempted to establish a ssh connection to the instance:
$ ssh -v -i ec2-keypair.pem ubuntu#ec2-174-129-185-190.compute-1.amazonaws.com
OpenSSH_5.1p1 Debian-5ubuntu1, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to ec2-174-129-185-190.compute-1.amazonaws.com [174.129.185.190] port 22.
debug1: Connection established.
debug1: identity file ec2-keypair.pem type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.1p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.1p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.1p1 Debian-5ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'ec2-174-129-185-190.compute-1.amazonaws.com' is known and matches the RSA host key.
debug1: Found key in /home/default/.ssh/known_hosts:11
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: ec2-keypair.pem
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).
What could be the problem and how to make it work?
For Ubuntu instances:
chmod 600 ec2-keypair.pem
ssh -v -i ec2-keypair.pem ubuntu#ec2-174-129-185-190.compute-1.amazonaws.com
For other instances, you might have to use ec2-user instead of ubuntu.
Most EC2 Linux images I've used only have the root user created by default.
See also: http://www.youtube.com/watch?v=WBro0TEAd7g
Now it's:
ssh -v -i ec2-keypair.pem ec2-user#[yourdnsaddress]
Canonical's releases use the user 'ubuntu' by default for anyone landing here with a ubuntu image that is coming up with the same problem.
If you're using a Bitnami image, log in as 'bitnami'.
Seems obvious, but something I overlooked.
For my ubuntu images, it is actually ubuntu user and NOT the ec2-user ;)
Ubuntu 10.04 with openSSH
this is the exact usage:
ssh -v -i [yourkeypairfile] ec2-user#[yourdnsaddress]
for example:
ssh -v -i GSG_Keypair.pem ec2-user#ec2-184-72-204-112.compute-1.amazonaws.com
above example was taken directly from the AWS tutorial for connecting to a Linux/UNIX machine at:
http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/
It will also complain if the pem file permissions are too open. chmod the file to 600 to fix that.
I was also running into this - turns out I was using a community-created AMI - and the default username was niehter root, nor was it ect-user or ubuntu. In fact, I had no idea what it was - till I tried 'root' and the server kindly asked me to login as xxx where xxx is whatever it tells you.
-cheers!
If you are running AWS image from Bitnami. The username would be bitnami. Cheers!
see my debug and look at the last one:
*
ssh -v -i awsliferaysrta.pem.txt root#54.254.250.***
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to 54.254.250.*** [54.254.250.***] port 22.
debug1: Connection established.
debug1: identity file awsliferaysrta.pem.txt type -1
debug1: identity file awsliferaysrta.pem.txt-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.1 pat OpenSSH_5*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 05:5c:78:45:c9:39:3a:84:fe:f8:19:5d:31:48:aa:5f
debug1: Host '54.254.250.***' is known and matches the RSA host key.
debug1: Found key in /Users/macbookpro/.ssh/known_hosts:2
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: awsliferaysrta.pem.txt
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to 54.254.250.*** ([54.254.250.***]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Remote: Port forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Forced command.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Please login as the user "bitnami" rather than the user "root".
*
You need have your private key in your local machine
You need to know the IP address or DNS name of your remote machine or server, you can get this from AWS console
If you are a linux user
Make sure the permissions on the private key are 600
(chmod 600 <path to private key file>)
Connect to your machine using ssh
(ssh -i <path to private key file> <user>#<IP address or DNS name of remote server>)
If you are a windows user
Use PuTTy to create the ssh session (http://the.earth.li/~sgtatham/putty/latest/x86/putty-0.66-installer.exe)
If your private key file is in .pem format convert it into .ppk using puttygen
Launch PuTTy, set you ppk file, IP address or DNS name of the remote server and start the ssh session
use...
# chmod 400 ec2-keypair.pem
don't use the 600 permission otherwise you might overwrite your key accidently.
For Debian EC2 instances, the user is admin.
There are 2 steps to be connected:
Chmod 400 on your private key, like this the others cannot access to your key:
chmod 400 toto.pem
To connect to your instance in SSH, you need to know the public IP address of your instance :
ssh -i toto.pem ec2-user#XX.XX.XX.XXX
Hope it helps !
this worked for me:
ssh-keygen -R <server_IP>
to delete the old keys stored on the workstation
also works with instead of
then doing the same ssh again it worked:
ssh -v -i <your_pem_file> ubuntu#<server_IP>
on ubuntu instances the username is: ubuntu
on Amazon Linux AMI the username is: ec2-user
I didn't have to re-create the instance from an image.
Permission for ec2-keypair.pem should be 400
chmod 400 ec2-keypair.pem
If you are using EBS, you can also try to mount the EBS Volume on a running instance. Then mount it on that running instance and see what's going on in /home. You can see things like is the user ubuntu or ec2-user ? or does it have the right public keys under ~/.ssh/authorized_keys
In my case (Mac OS X), the problem was the file's break type. Try this:
1.- Open the .pem file with TextWrangler
2.- At Bottom of app, verify if the Break Type is "Windows(CRLF)".
Its ec2-user for Amazon Linux AMI's and ubuntu for Ubuntu images.
Also, RHEL 6.4 and later ec2-user
RHEL 6.3 and earlier root
Fedora ec2-user
Centos root
Just adding to this list. I was having trouble this morning with a new user just added to an AWS EC2 instance. To cut to the chase, the problem was selinux (which was in enforcing mode), together with the fact that my user home dir was on a new EBS attached volume. Somehow I guess selinux doesn't like that other volume. Took me a while to figure out, as I looked through all the other usual ssh issues (/etc/ssh/sshd_config was fine, of course no password allowed, permissions were right, etc.)
The fix?
For now (until I understand how to allow a user to ssh to a different volume, or somehow make that volume a bona fide home dir point):
sudo perl -pi -e 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
sudo setenforce 0
That's it. Now my new user can log in, using his own id_rsa key.
Had the same issue. Permission denied (publickey) when trying to login in with 'ec2-user' or with 'root'.
Googled the AMI number of the machine image and it had the SSH login information right their on the Debian wiki page.
Hope this helps.