When following the tutorial instructions for connecting to my JobFlow in EMR, I type following:
./elastic-mapreduce --jobflow j-3FLVMX9CYE5L6 --ssh
and get this error:
Permission denied (publickey)
I'm already able to run other elastic-mapreduce commands just fine to create flows etc, so I'm assuming there's security settings required on the actual master instance for the flow, but nothing in the tutorial explains how to configure this (after all, I need to SSH into it to do the configuration in the first place!)
I found that I need to login as user "hadoop" using the EC2 keypair, and not any of the regular suspects (ec2-user, root, etc.) Like:
ssh -i privatekey.pem hadoop#masternode
Hope this is useful to someone.
Ok now I feel sheepish: I was using the Amazon CloudFront keypair from the my initial account setup rather than keypair associated with my account for accessing EC2 instances, accessible from EC2 > Network & Security > Key Pairs in the AWS Management Console.
The command "ssh -i privatekey.pem hadoop#masternode" worked great. The user "hadoop" must be used for "ec2 elastic mapreduce".
Related
I am new to AWS and I have created a new Windows EC2 instance. I see ways to SSH to different instances like Amazon Linux, Ubuntu using PuTTY. But not for Windows instance. Can we not connect to Windows instance using PuTTY? Any help would be appreciated. Thanks.
You can SSH to Linux and Mac instance on AWS. You can use Remote Desktop (RDP) to access Windows Instances.
Suppose you wanted to SSH into a Windows instance, then you would have to install OpenSSH. Which would still require access to the Windows instance.
AWS offers a few ways to access your Windows instance, one of them is AWS Systems Manager Session Manager (SSM). You would need to enable RDP through SSM. You can find out how to do this via this workshop. Alternatively configure SSM to run powershell commands against the instance Note the instance must have a role that has IAM access to SSM. If the instance is in a private VPC it would require access to SSM via endpoints.
If your instances is in a public subnet, and has a public IP address then you can easily connect to your Windows instance using RDP. Note just like your EC2 Instances running Linux, the security group must allow access. In this case it must be to RDP (3389) and not SSH (Port 22). You can read more about the prerequisites here.
When you created your Windows EC2 instance, you were asked to create a key. You can use this key to get the Windows password, assuming you used an AWS AMI.
Access your password
Follow these steps (From the documentation here):
Open the Amazon EC2 console, and then choose Instances.
Select the check box for the instance, and then expand the Actions dropdown list. If you're using the old console, then choose Get Windows Password. If you're using the new console, choose Security, and then choose Get Windows Password.
Note: It can take a few minutes for this option to be available after you first launch a new instance.
Choose Browse, select your key pair file, and then choose Open.
-or-
Paste the contents of your key pair into the text box.
Choose Decrypt Password.
Connect to the Instance
On the password screen, Choose Download remote desktop file. Your browser prompts you to either open or save the RDP shortcut file. Select the option to save the file. When you have finished downloading the file, choose Cancel to return to the Instances page.
Navigate to your downloads directory and open the RDP shortcut file.
You might get a warning that the publisher of the remote connection is unknown. Choose Connect to continue to connect to your instance.
The administrator account is chosen by default. Copy and paste the password that you saved previously.
You can also use EC2 Serial Console to access an EC2 instance running Windows, this is great to debug boot issues.
Due to the nature of self-signed certificates, you might get a warning that the security certificate could not be authenticated. Use the following steps to verify the identity of the remote computer, or simply choose Yes (Windows) or Continue (Mac OS X) if you trust the certificate.
For more details on how to connect, visit this site.
Can we SSH to Windows EC2 instance in AWS? - NO.
Windows EC2 you can connect using RDP( Remote Desktop ). You can download .rdp file from AWS console navigating through select instance -> connect -> RDP
SSH connection is for Linux based EC2 machine.
Late here but the answer is Yes you can SSH into a Windows EC2 instance from Windows/Linux and Mac(haven't tried). The upshot is that you need an SSH server in order to receive ssh requests from clients.
Make sure your Windows instance is running
RDP into the instance (one-time only)
Type powershell in command line to toggle out of cmd
Run the following command to determine if you have OpenSSH.Server installed
Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'
Install the service
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
After a minute or two it installs (be patient)
Set service to start service automatically in case you stop instance
Set-Service -Name sshd -StartupType 'Automatic'
Run the OpenSSH.Server service, called sshd
Start-Service sshd
Exit the instance shell and RDP session
Go back to your shell on your computer
Run your customary ssh command to get into the EC2 instance. You'll be prompted for a password. There are ways to get around that.
Yes, recent releases of Windows (10 build 1809, Server 2019, and later) offer official support for a native OpenSSH daemon. See https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse?tabs=powershell for details about OpenSSH on Windows.
When wanting to use SSH to connect to an EC2 instance specifically, I have found that the easiest approach is to build a new AMI with the OpenSSH package preinstalled and the relevant services preconfigured. The full process that is currently working for me:
Build a Windows AMI based on Server 2019 or later (e.g. use Windows_Server-2019-English-Full-ECS_Optimized-2022.12.14 as the base AMI). As part of that AMI:
Install OpenSSH and configure the sshd and ssh-agent services as described in the above link:
$ErrorActionPreference = 'Stop'
Write-Host 'Installing and starting sshd'
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
Set-Service -Name sshd -StartupType Automatic
Start-Service sshd
Write-Host 'Installing and starting ssh-agent'
Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0
Set-Service -Name ssh-agent -StartupType Automatic
Start-Service ssh-agent
Write-Host 'Set PowerShell as the default SSH shell'
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value (Get-Command powershell.exe).Path -PropertyType String -Force
Configure PowerShell as the default SSH shell:
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value (Get-Command powershell.exe).Path -PropertyType String -Force
Launch an EC2 instance using the new AMI:
Provide a valid, existing SSH keypair.
Select IMDSv2.
Provide the following PowerShell script as the userdata script. This script will ensure the SSH keypair specified when launching will be added to the the Administrator user's authorized keys file. Note that the <powershell> and </powershell> tags are part of the userdata; they are parsed and extracted by AWS prior to the script being executed.
<powershell>
# Userdata script to enable SSH access as user Administrator via SSH keypair.
# This assumes that
# 1. the SSH service (sshd) has already been installed, configured, and started during AMI creation;
# 2. a valid SSH key is selected when the EC2 instance is being launched; and
# 3. IMDSv2 is selected when launching the EC2 instance.
# Save the private key from instance metadata.
$ImdsToken = (Invoke-WebRequest -Uri 'http://169.254.169.254/latest/api/token' -Method 'PUT' -Headers #{'X-aws-ec2-metadata-token-ttl-seconds' = 2160} -UseBasicParsing).Content
$ImdsHeaders = #{'X-aws-ec2-metadata-token' = $ImdsToken}
$AuthorizedKey = (Invoke-WebRequest -Uri 'http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key' -Headers $ImdsHeaders -UseBasicParsing).Content
$AuthorizedKeysPath = 'C:\ProgramData\ssh\administrators_authorized_keys'
New-Item -Path $AuthorizedKeysPath -ItemType File -Value $AuthorizedKey -Force
# Set appropriate permissions on administrators_authorized_keys by copying them from an existing key.
Get-ACL C:\ProgramData\ssh\ssh_host_dsa_key | Set-ACL $AuthorizedKeysPath
# Ensure the SSH agent pulls in the new key.
Set-Service -Name ssh-agent -StartupType "Automatic"
Restart-Service -Name ssh-agent
</powershell>
Connect to the instance via SSH like normal. Provide the SSH keypair you specified when launching and user Administrator. For example:
ssh -i ~/.ssh/my-keypair Administrator#my.ec2.instance
PuTTy is not the tool to use typically for Windows machines. While I'm sure it is possible to setup SSH access for Windows machines, the methods below are more typical and easier to setup.
If you have a keypair associated to your EC2 instance, you can use the AWS console to show you the username and password needed to use Windows Remote Desktop to access your machine by giving it your pem file. This will also require you to enable inbound RDP access (a different port than SSH) in your security group.
If you just want PowerShell access and you are using an instance with with the SSM agent installed (e.g. Amazon Linux AMIs) and a role with the required SSM permissions, you can use Session Manager to connect. This does not need a keypair or direct network access to your machine so this is a more secure method but requires a bit more setup.
Both techniques are described is more detail in the reference below.
References:
https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/connecting_to_windows_instance.html
You can connect to your Windows EC2 via AWS Session Manager. It let you to manage your instance from browser based shell. Like SSH, you can manage your EC2 from the shell provided by the SSM. You can execute power shell commands from the console.
To use SSM, first you need to install ssm agent in instance and grant EC2 with required SSM policies in IAM role.
I believe SSM agent is by default installed in Windows EC2 instances. Also AWS providing one role named AmazonSSMRoleForInstancesQuickSetup for EC2. You can use the same or utilise the policies inside this role.
To connect: Select the instance and in the Connect option select Session manager
I have created an ec2-intance on AWS. But when im trying to connect to it by using my .pem file, im getting error message: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). I have changed the permission to this file by chmod 400 myfile.pm.
This is the command i use to connect to my instance: ssh -i ec2demo.pem ec2demo#ec2-35-158-140-25.eu-central-1.compute.amazonaws.com
I also searched for the issue on internet, and some people say i need to type chmod 600 myfile.pem. It still not works. Im using macOS Mojave, and the ssh client integrated. Do i need to install the AWS-CLI to make it works? Or should it work without AWS-CLI? And is it better to use ssh client from homebrew, or?
Thanx for help
When launching a new Amazon Linux instance on Amazon EC2, the public half of the keypair is copied to:
/users/ec2-user/.ssh/authorized_keys
You can then login to the instance using the private half of the keypair:
ssh -i key.pem ec2-user#1.2.3.4
(Or, you can use a DNS name instead of an IP address.)
It sounds like you have not logged into this instance yet, so make sure you login as ec2-user instead of ec2demo. The name of the instance does not impact the Linux user on the instance.
I'm trying to run a Spark cluster on AWS using https://github.com/amplab/spark-ec2.
I've generated a key and and login credentials, and I'm using this command:
./spark-ec2 --key-pair=octavianKey4 --identity-file=credentials3.csv --region=eu-west-1 --zone=eu-west-1c launch my-instance-name
However, I keep getting this:
Warning: SSH connection error. (This could be temporary.)
Host: mec2-myHostNumber.eu-west-1.compute.amazonaws.com
SSH return code: 255
SSH output: Warning: Permanently added 'ec2-myHostNumber.eu-west-1.compute.amazonaws.com,myHostNumber' (ECDSA) to the list of known hosts.
Permission denied (publickey).
If I quit the console and then try to start the cluster again, I get this:
Setting up security groups...
Searching for existing cluster my-instance-name in region eu-west-1...
Found 1 master, 1 slave.
ERROR: There are already instances running in group my-instance-name-master or my-instance-name-slaves
The command is incorrect. Key pair name should be the one you mention in AWS. Identity file is .pem file associated. You can't ssh into a machine with AWS credentials (your csv file is credentials).
./spark-ec2 --key-pair=octavianKey4 --identity-file=octavianKey4.pem --region=eu-west-1 --zone=eu-west-1c launch my-instance-name
Can you add --resume to your spark-ec2 command and try? Your slave may not have the key. --resume will make sure it is transferred to the slave.
Running Spark on EC2
If one of your launches fails due to e.g. not having the right
permissions on your private key file, you can run launch with the
--resume option to restart the setup process on an existing cluster.
This is an extension of my last question considering I've decided to deploy a Docker container onto a ton of EC2's. I've set up a repository and a user with full rights, and I added the correct keys to my aws cli configuration. When I try to run the docker login command that comes up after running the "aws ecr get-login" command, it gives me a failed with status: 403 forbidden error. I have absolutely no clue what's going on, and I've spent the past 2 days trying to fix this error... Any ideas?
I would suggest to check the security group of the EC2 Instance
To allow access via SSH you have to apply the following settings for the Security Group of the EC2 Instance:
Security Groups
I am creating an aws ec2 instance using this tutorial, and I can't find any information on troubleshooting my issue, or any evidence that anyone else has even experienced this!
I used an IAM user with admin permissions to set up an ec2 instance, and when I run
$> aws ec2 get-console-output --instance-id <my-ec2-id>
a blank line is output, followed by
'Output'
and nothing else!
According to the tutorial, this command would enable me to see the remote RSA fingerprint to verify I'm making the right connection.
I can log into my ec2 instance just fine (though I suppose without the previous step there's no way to be absolutely sure).
Additionally, the IAM user I'm working with is not my CLI's default user, and I set up a profile to handle it. But if I try
$> aws ec2 get-console-output --profile <user-profile> --instance-id <my-ec2-id>
I still get the same results as before. The maddening thing is that I have solved this problem before, but I can't remember how.
Certain AWS CLI operations may not explicitly state if the credentials are invalid or if users are lacking the roles/permissions to access the resources defined. In this case, it is likely due to the Access Credentials being invalid - and you can verify this with a describe-instances or similar command.
In older versions of the CLI (~1.7), in order to easier debug this, you can use the --debug argument, such as:
> aws ec2 get-console-output --instance-id i-<id> --debug
<Errors><Error><Code>InvalidInstanceID.NotFound</Code><Message>The instance ID 'i-e7bffa43' does not exist</Message></Error></Errors>
In newer versions of the CLI (1.9) this particular argument gives a bit more detail in its error:
> aws ec2 get-console-output --instance-id i-<id>
A client error (InvalidInstanceID.NotFound) occurred when calling the GetConsoleOutput operation: The instance ID 'i-<id>' does not exist