AWS CLI is problematic from within an EC2 instance.
I am on an EC2 Instance via Putty as well as Windows PowerShell (at different times). Command is: aws ec2 describe-instances
Result & error is:
Hanging and Could not connect to the endpoint URL: "https://ec2.us-east-1.amazonaws.com/"
The AWS CLI command is OK and correctly replies w json output when the CLI command executed from my local windows command prompt.
'aws configure' seems OK with the region is us-east-1
The EC2 instance can ping 8.8.8.8 as well as another instance (in same VPC, SubNet, region, etc) with a public IP.
I think I allowed everything open in the Security Groups (for troubleshooting purposes):
img of the security group
Network ACLs also left alone:
enter image description here
These are new and blank EC2 instances. To also note: 'sudo yum update' also returns an error of:
Could not retrieve mirrorlist http://repo.us-east-1.amazonaws.com/latest/main/mirror.list error was
12: Timeout on http://repo.us-east-1.amazonaws.com/latest/main/mirror.list: (28, 'Resolving timed out after 5515 milliseconds')
Thought I have all connections open (for sake of troubleshooting). Possible clue: Read elsewhere that my internet connection has to be "legit." I am using a free hotel wifi. But this doesnt make sense as its the EC2 doing the pinging, and my windows cmd can return results.
Thank you very much.
-Martin
Related
I'm trying to access lambda functions from a Windows VM I have created in EC2 for dev purposes but even a simple 'list functions' command fails to connect
I have tried using the AWS CLI through PowerShell, the dotnet sdk and the VS AWS Toolkit but each of these times out after a long waiting period. I can, however, list other services such as my databases and S3 buckets.
aws cli failure message
VS toolkit failure message
I have tried creating a new VM with the same results. I've disabled windows firewall altogether, allowed all traffic through the security group and have VPC endpoints for my subnet (ssm, ec2messages, lambda, ec2).
I have no trouble connecting to the lambda service through my own computer. On the VM, I have modified the .aws/credentials file to match the one on my computer for both the admin and current user but I still can't connect. This tells me that the problem isn't related to my access key credentials.
I'm reaching the end of the troubleshooting options I can think of so any help would be very much appreciated!
Update: using telnet, I cannot connect to lambda.ap-southeast-2 but I can connect to s3.ap-southeast-2 and lambda.ap-southeast-1. It seems lambda.ap-southeast-2 is being blocked somewhere but it isn't windows firewall because it's off and the same problem happens on Ubuntu VMs.
In the VPC Management Console, I haven't set up any firewalls under network or dns filewalls and my network ACL allows all traffic.
Launching an AWS EC2 instance seems quite straightforward although when it comes to connecting to the newly launched instance things get sticky. The process for connecting to an instance proposed by such a tech giant is very counter-intuitive.
As a short reminder I should add that an "instance" is technically a virtual machine running on Amazon's Elastic Compute Cloud (EC2), for more info one could have a look at this link.
The ec2 instance referred to in this discussion is Ubuntu Server 20.04 LTS (HVM).
The instruction for working with EC2 Linux instances is given here.
AWS EC2 proposes three different ways of connecting to an instance:
EC2 Instance connect (browser-based SSH connection),
Session Manager
SSH Client
Now with regard to connecting to the above-mentioned instance there are only certain connections that establish correctly and the rest of the proposed methods fail, here is the list of connection successes and failures :
Ubuntu instance, security group source "Custom=0.0.0.0/0", Connection establishes using both EC2 Instance Connect (browser-based SSH connection) and SSH client.
Ubuntu instance, security group source "My IP=$IP", Connection establishes only using SSH client (terminal on Ubuntu and PuTTY on windows) and not using EC2 instance connect.
Both above cases have been tried on Ubuntu 20.04 and Windows 10 as local machine and the problem remains similar on both machines. I went through most of the failure cases discussed in the troubleshooting documents proposed here and verified them on my instance. Yet the problem persists. I should also add that I never tried "session manager" connection method although opening its tab already would give some info about "not installed" agents and features.
Any idea regarding this problem? Somebody out there facing the same issue?
From Docs
(Amazon EC2 console browser-based client) We recommend that your instance allows inbound SSH traffic from the recommended IP block published for the service.
Reason for this -> EC2 Instance Connect works by making an HTTPS connection between your web browser and the backend EC2 Instance Connect service on aws. Then, EC2 Instance Connect establishes a "mostly normal" SSH connection to the target instance in other words the request is going from backend ec2 instance connect and not your browser that is why it needs IP address from accepted ranges of that region .
Browser based EC2 Instance Connect uses specific IP ranges for browser-based SSH connections to your instance. These IP ranges differ between AWS Regions. To find the AWS IP address range for EC2 Instance Connect in a specific Region, use the following( just replace your region with your region) ( for Linux required curl and jq as prerequisite)
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json| jq -r '.prefixes[] | select(.region=="Your region") | select(.service=="EC2_INSTANCE_CONNECT") | .ip_prefix'
whatever the value is returned just add up to your security rule and it will work.
Ubuntu instance, security group source "Custom=0.0.0.0/0", Connection establishes using both EC2 Instance Connect (browser-based SSH connection) and SSH client.
this works because 0.0.0.0/0 allows connection from all the IP ranges( which includes your region IP too).
for more details try reading this troubleshoot
I have a file on a server located at let's say: https://example.com/myfolder/file.gz
The domain example.com is protected by inbound rules such that I can only access it by connecting to a VPN.
So on my personal hardware when I try to download the file by using curl -O https://example.com/myfolder/file.gz, it gives me "Unknown host error". However, when I connect to VPN via Cisco AnyConnnect and then hit the curl command, file.gz gets downloaded.
Now what I want is to enable an AWS EC2 instance to be able to download file.gz by executing curl command. However, when I SSH into my EC2 instance and execute the curl command, it gives me the same "Unknown host error" as expected.
I have searched around VPC, ENI, OpenVPN and other AWS offerings but since I'm a beginner to AWS I didn't quite understand what could be the solution to my problem.
FYI - One way to possibly resolve this issue would be to create an elastic IP and whitelist that in the server that I'm trying to access, (in this case example.com) but I want to create more EC2 instances trying to do the same thing, so it would become tedious to manage.
About 6 months ago I created an AWS EC2 instance to mess around with on the free tier. After months of having no issues remoting into my AWS EC2 server, I've recently been unable to access it via SSH. I am using the following command:
ssh -i my-key-pair.pem ec2-user#ec2-**-**-***-***.us-****-*.compute.amazonaws.com
...and after a minute or two, am getting this response
ssh: connect to host ec2-**-**-***-***.us-****-*.compute.amazonaws.com port 22: Operation timed out
What's strange is that
1) I can read and write to my RDS database just fine
2) I can ping into the server
3) My port 22 is open
4) The instance is running and healthy
5) In the Inbound section of the security group of the EC2 server it allows for all traffic and SSH from any location via port 22.
6) I'm using the same key-pair as always
I went through this documentation (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.html) and can confirm that the VPC, subnet, network ACL and route tables all line up (I haven't changed anything with those since the SSH stopped working). Any insight would be extremely helpful!
Sometimes the instance fails, you can check the screen of it via AWS
console.
Run another instance in the same security group and try to
connect to it and then from there to your original one - to verify if
ssh is still open (even if you do not have the ssh key, the error
will not be 'timeout')
You can create a snapshot of your instance and
attach it as another volume in a new one and you can investigate
logs, maybe something went wrong.
You can restart the instance, if
for example i ran out of memory it will most likely work after the
reboot (hopefully for a long enough time for you to investigate).
You can contact AWS support.
I'm writing a flask API in pycharm. When I run my code locally, requests using boto3 to get secrets from secrets manager take less than a second. However, when I put my code on an EC2, it takes about 3 minutes (tried in both t2.micro and m5.large).
At first I thought it could be a Python issue, so I ran it in my EC2s through the awscli using:
aws secretsmanager get-secret-value --secret-id secretname
It sill took about 3 minutes. Why does this happen? Shouldn't this in theory be faster in an EC2 than in my local machine?
EDIT: This only happens when the EC2 is inside a VPC different than the default VPC.
After fighting with this same issue on our local machines for almost two months, we finally had some forward progress today.
It turns out the problem is related to IPv6.
If you're using IPv6, then the secrets manager domain will resolve to an IPv6 address. For some reason the cli is unable to make a secure connection using IPv6. After it times out, the cli falls back to IPv4 and then it succeeds.
To verify if you're resolving to an IPv6 address, just ping secretsmanager.us-east-1.amazonaws.com. Don't worry about the ping response, you just want to see the IP address the domain resolves to.
To fix this problem, you now have 3 options:
Figure out your networking issues. This could be something on your machine or router. If in an AWS VPC, check your routing tables and security groups. Make sure you allow outbound IPv6 traffic (::/0).
Reduce the cli connect timeout to make the IPv6 call fail faster. This will make the IPv4 fallback happen sooner. You may want give a better timeout value, but the general idea is to add something like this: --cli-connect-timeout 1
Disable IPv6. You can either disable IPv6 on your machine/router altogether, or you can adjust your machine to prefer IPv4 for this specific address (See: https://superuser.com/questions/436574/ipv4-vs-ipv6-priority-in-windows-7).
Ultimately, option 1 is the real solution, but since it is so broad, the others might be easier.
Hopefully this helps someone else maintain a bit of sanity when they hit this.
I had this issue when working from home through the Cisco AnyConnect VPN client. Apparently it blocks anything IPv6.
The solution for me was to disable IPv6 altogether on my laptop.
To do so for macos:
networksetup -setv6off Wi-Fi # wireless
networksetup -setv6off Ethernet # wired
To re-enable:
networksetup -setv6automatic Wi-Fi # wireless
networksetup -setv6automatic Ethernet # wired
I ran the following commands from my own computer and from an Amazon EC2 t2.nano instance in the ap-southeast-2 region:
aws secretsmanager create-secret --name foo --secret-string 'bar' --region ap-southeast-2
aws secretsmanager get-secret-value --secret-id foo --region ap-southeast-2
aws secretsmanager delete-secret --secret-id foo --region ap-southeast-2
In both cases, each command returned within a second.
Additional:
To test your situation, I did the following (in the Sydney region):
Created a new VPC using the VPC Wizard (just a public subnet)
Launched a new Amazon EC2 instance in the new VPC, with a Role granting permission to access Secrets Manager
Upgraded AWS CLI on the instance (the installed version didn't know about secretsmanager
Ran the above commands
They all returned immediately.
Therefore, the problem lies with something to do with your instances or your VPC.
I made the hotspot from my phone and it worked