I have an EC2 instance in a private subnet in which I want to copy files.
Instead of a S3 bucket I want to use Secure File Copy through Session Manager as documented on here and announced on here.
A running EC2 instance is attached with an instance profile containing the policy AmazonEC2RoleforSSM. On my local machine (macOS 10.14.5) the AWS CLI (aws-cli/1.16.195) and the Session Manager Plugin (1.1.26.0) is installed and .ssh/config is configured accordingly.
I can log into the instance with Session Manager on the web AWS Console.
I can log into the instance using the CLI with aws ssm start-session --target i-XXX.
I can't log into the instance using SSH. I've tried 2 different OpenSSH client versions:
OpenSSH_7.9p1:
When I run ssh ec2-user#i-XXX it hangs infinitely. However I can see an connected session in the Session Manager. When I SIGTERM the process I get following output and the session is terminated:
Command '['session-manager-plugin', '{"SessionId": "XXX", "TokenValue": "XXX", "StreamUrl": "wss://ssmmessages.eu-central-1.amazonaws.com/v1/data-channel/XXX?role=publish_subscribe", "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "RequestId": "XXX", "HTTPHeaders": {"x-amzn-requestid": "XXX", "date": "Wed, 07 Aug 2019 08:47:23 GMT", "content-length": "579", "content-type": "application/x-amz-json-1.1"}}}', 'eu-central-1', 'StartSession', u'cc', '{"DocumentName": "AWS-StartSSHSession", "Target": "i-XXX", "Parameters": {"portNumber": ["22"]}}', u'https://ssm.eu-central-1.amazonaws.com']' returned non-zero exit status -13
OpenSSH_8.0p1:
When I run ssh ec2-user#i-XXX I get the following error and need to manually terminate the session in the Session Manager:
kex_exchange_identification: banner line contains invalid characters
I just got an answer from AWS Support and it working for me now. There was a bug in one of the following components.
Ensure at least following versions and it should work then.
local
aws cli: aws-cli/1.16.213 Python/3.7.2 Darwin/18.7.0 botocore/1.12.203
aws --version
session-manager-plugin: 1.1.26.0
session-manager-plugin --version
target ec2 instance
amazon-ssm-agent: 2.3.687.0
for AmazonLinux yum info amazon-ssm-agent | grep "^Version"
I've also created a neat SSH ProxyCommand script that temporary adds your public ssh key to target instance during connection to target instance.
AWS SSM SSH ProxyComand -> https://gist.github.com/qoomon/fcf2c85194c55aee34b78ddcaa9e83a1
Related
I'm dealing with an EC2 instance which suddenly stopped to work via SSM (and a reboot is not fixing it).
The keypair is missing, so no access via SSH.
The only way to access would be using EC2 Serial Console. Using the web client, the screen remains black, so instead, I'm using the alternative version of pushing my key.
The action to push the key is successful.
$ aws ec2-instance-connect send-serial-console-ssh-public-key --instance-id i-123456abcd --serial-port 0 --ssh-public-key file://test.pub --region us-east-1
{
"RequestId": "dsadasdasdasd",
"Success": true
}
However, when I try to SSH, I'm getting a wrong publickey error.
$ ssh -i test i-123456abcd.port0#serial-console.ec2-instance-connect.us-east-1.aws
i-123456abcd.port0#serial-console.ec2-instance-connect.us-east-1.aws: Permission denied (publickey)
For testing's sake, I create a new pair, same error.
Serial Console access is enabled account-wide and my role has AdministratorAccess.
Any ideas about how to proceed?
Hi need to transfer a file to ec2 machine via ssm agent. I have successfully installed ssm-agent in ec2 instances and from UI i am able to start session via "session-manager" and login to the shell of that ec2 machine.
Now I tried to automate it via boto3 and using the below code,
ssm_client = boto3.client('ssm', 'us-west-2')
resp = client.send_command(
DocumentName="AWS-RunShellScript", # One of AWS' preconfigured documents
Parameters={'commands': ['echo "hello world" >> /tmp/test.txt']},
InstanceIds=['i-xxxxx'],
)
The above works fine and i am able to send create a file called test.txt in remote machine but his is via echo command
Instead I need to send a file from my local machine to this remove ec2 machine via ssm agent, hence I did the following ,
Modified the "/etc/ssh/ssh_config" with proxy as below,
# SSH over Session Manager
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
Then In above code, I have tried to start a session with below code and that is also successfully .
response = ssm_client.start_session(Target='i-04843lr540028e96a')
Now I am not sure how to use this session response or use this aws ssm session and send a file
Environment description:
Source: pod running in an EKS cluster
dest: ec2 machine (which has ssm agent running)
file to be transferred: Important private key which will be used by some process in ec2 machine and it will be different for different machine's
Solution tried:
I can push the file to s3 in source and execute ssm boto3 libaray can pull from s3 and store in the remote ec2 machine
But I don't want to do the above due to the reason I don't want to store the private key i s3. So wanted to directly send the file from memory to the remote ec2 machine
Basically i wanted to achieve scp which is mentioned in this aws document : https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html#sessions-start-ssh
If you have SSH over SSM setup, you can just use normal scp, like so:
scp file.txt ec2-user#i-04843lr540028e96a
If it isn't working, make sure you have:
Session Manager plugin installed locally
Your key pair on the instance and locally (you will need to define it in your ssh config, or via the -i switch)
SSM agent on the instance (installed by default on Amazon Linux 2)
An instance role attached to the instance that allows Session Manager (it needs to be there at boot, so if you just attached, reboot)
Reference: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html
If you need more detail, give me more info on your setup, and I'll try and help.
I'm trying to figure out what version of Instance Metadata Service my ec2 instance is using.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#configuring-instance-metadata-options
If you want to determine it from the EC2 instance, you can just try sending a request to http://169.254.169.254/ and see what the status code is.
For example, this instance has IMDSv2 enabled and requests without a token are not accepted:
$ curl -w "%{http_code}\n" http://169.254.169.254/
401
The 401 status code means Unauthorized.
If you have AWS access keys with permissions to describe EC2 instances, then you can run the following:
$ aws ec2 describe-instances --region us-west-2 --instance-id i-0123456789abcdef --query "Reservations[0].Instances[0].MetadataOptions"
{
"State": "applied",
"HttpTokens": "optional",
"HttpPutResponseHopLimit": 1,
"HttpEndpoint": "enabled"
}
This server does not require IMDSv2 (HttpTokens is optional).
To enable IMDSv2, you can run aws ec2 modify-instance-metadata-options. See more in AWS documentation on configuring the instance metadata options.
Let me summarise what I found here
Connect to an instance via the EC2 service in the AWS Console
Put the IMDSv1 command: curl http://169.254.169.254/latest/meta-data/
If you receive a list items then your instance can use IMDSv1 requests. If you receive 401 - Unauthorized then it uses IMDSv2 or non.
Now put the IMDSv2 command: TOKEN=curl -X PUT "<http://169.254.169.254/latest/api/token"> -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/
If you receive a list of items, then your instance can use IMDSv2 requests. Otherwise, your instance is not allowed to request meta data at all (regardless of the version)
If you want to change this configuration follow this.
I need to pass "sudo su - < user >" command once i logged though AWS CLI using aws ssm?
aws ssm start-session --target "instance ID" ??????? "sudo su - < user >"
Is there any way? Passing as parameters or something?
It is possible to get into an EC2 instance from the command line without SSH. Whenever you can avoid using SSH, and use more cloud-native approaches such as System Manager's Session Manager, that is recommended.
The documentation says:
Example 1: To start a Session Manager session
This start-session example establishes a connection with an instance
for a Session Manager session. Note that this interactive command
requires the Session Manager plugin to be installed on the client
machine making the call.
aws ssm start-session \
--target "i-1234567890abcdef0"
Output:
Starting session with SessionId: Jane-Roe-07a16060613c408b5
So you can get into your EC2 instance that way, and then enter "sudo su - < user >".
However, passing in parameters with the aws ssm start-session AWS CLI command is currently not supported, as that same documentation page says:
--parameters (map)
Reserved for future use.
key -> (string)
value -> (list)
(string)
I'm trying to ssh to my docker container in Elastic Beanstalk via awsebcli. But it always complains Access Denied by the following command:
eb init --profile default1 Microservices-Docker --debug
The debug info is listed below:
2017-04-06 10:15:29,724 (DEBUG) ebcli.lib.aws : Making api call: (elasticbeanstalk, describe_configuration_settings) to region: eu-west-1 with args:{'ApplicationName': 'Microservices-Docker', 'EnvironmentName': 'customer-track-test'}
2017-04-06 10:03:41,541 (DEBUG) ebcli.lib.aws : Response: {'ResponseMetadata':
{'date': 'Thu, 06 Apr 2017 06:03:41 GMT', 'RetryAttempts': 0,
'HTTPStatusCode': 403, 'RequestId': 'c8bce47d-1a8e-1117-92b2-b348ebce885e'},
'Error': {'Message': 'Access Denied', 'Code':
'InsufficientPrivilegesException', 'Type': 'Sender'}}
2017-04-06 10:03:41,541 (DEBUG) ebcli.lib.aws : API call finished, status = 403
2017-04-06 10:03:41,541 (DEBUG) ebcli.lib.aws : Received a 403
NotAuthorizedError: Operation Denied. Access Denied
I add full access policy to Elastic Beanstalk to my current user, what could possibly cause that error, any suggestions? Thanks!
eb init does not ssh into a docker container, it initializes an elastic beanstalk configuration within the current directory you are working on. You probably don't have the privileges to do that in your current location, try a directory you have full control over.
See more in the documentation - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-cmd-commands.html#eb3-cmd-options
In order to ssh into the docker container you must first ssh into the EC2 instance running your container using eb ssh (or any other method), and then ssh into the docker container running there using docker exec or docker attach
-- EDIT --
Reading through your errors again I see the error does originate from AWS. Are you sure your CLI is configured to use the user you set the policies for? You can check that by looking at the access key configured in ~/.aws/configure