I am encountering an issue where I am unable to receive a response when attempting to ping from a namespace located within a public EC2 instance to a private one. To better illustrate my situation, I have included a diagram below.
Related
I am trying to access Timestream from EC2/Lambda instances that run within a VPC so that I can speak to a RDS instance from those EC2 instances/Lambda functions. I have spent many hours trying to get access to Timestream via PrivateLink/a VPC instance endpoint to work and think I may have found an issue. When I provision a VPC endpoint for the Timestream ingest service, the Private DNS name is specific to the cell endpoint, e.g. ingest-cell2.timestream.us-east-1.amazonaws.com NOT the general endpoint URL that boto3 uses, i.e. ingest.timestream.us-east-1.com. When I run a nslookup on ingest-cell2.timestream.us-east-1.amazonaws.com it properly resolves to the private IP of the VPC endpoint ENI, but if I lookup the more general endpoint URL of ingest.timestream.us-east-1.com it continues to resolve to public AWS IPs. The result of this is that if I initialize the timestream write client normally and perform any actions, it hangs because it is trying to communicate with a public IP from a private subnet,
import boto3
ts = boto3.client('timestream-write')
ts.meta.endpoint_url # https://ingest.timestream.us-east-1.amazonaws.com
ts.describe_endpoints() # hangs
ts.describe_database(DatabaseName='dbName') # hangs
If I explicitly give it the cell specific endpoint URL, the describe_endpoints() function throws an error but seemingly normal functions work (haven't tested writes or reads yet, just describing databses)
import boto3
ts = boto3.client('timestream-write', endpoint_url='https://ingest-cell2.timestream.us-east-1.amazonaws.com')
ts.describe_endpoints() # throws UnknwonOperationException error
ts.describe_databse(DatabaseName='dbName') # Succeeds
If I provision a NAT gateway in the private subnet rather than a VPC endpoint everything works normally as expected. Furthermore for fun, I tried adding the VPC endpoint private IP to the /etc/hosts file with ingest.timestream.us-east-1.com to force proper resolution and even then I get the same hanging behavior when running the above block of code
This seems pretty broken to me. The whole point of the VPC endpoint is to enable the SDK to operate normally. Maybe I am missing something?
I have an EC2 instance running inside an ECS cluster with VPC.
On the instance, I need to run a ECS task that needs access to DynamoDB.
When I try running the same task using Fargate, I can use the assignPublicIp = 'ENABLED' option to allow my task to have access to other AWS services, and everything works fine.
However, the assignPublicIp option is not available for the EC2 launch type, and I cannot figure out how to allow my EC2 instance have access to other AWS services.
I read the AWS docs and followed guides like this one to setup VPC endpoint for DynamoDB.
I also made sure that there aren't any network access restrictions by making sure that inbound/outbound rules for my NACL and security group for the VPC are wide open (at least for the sake of testing).
Here is how the rules look like, for both NACL and my security group:
Finally, I used the VPC > Reachability Analyzer to check if AWS can detect any problems regarding the connection path between my EC2 instance and DynamoDB, but the analysis reported a Reachable status.
It basically told me that there was no issues regarding establishing a connection along the following path:
Network interface for my EC2 instance (source)
Security group for the VPC
NACL for the VPC
Route table for the VPC
which includes the following route added by the VPC endpoint for DynamoDB
Destination: pl-02cd2c6b (com.amazonaws.us-east-1.dynamodb, 3.218.182.0/24, 3.218.180.0/23, 52.94.0.0/22, 52.119.224.0/20)
Target: the endpoint ID (e.g., vpce-foobar)
VPC endpoint for DynamoDB (destination)
Despite AWS telling me that I have a "Reachable" status, I still think it might be a network reachability problem, because when I run the task, the script I am running gets stuck right after it makes a GetItem call to DynamoDB.
If it was a permission error or an invalid parameter issue, I would get an error immediately, but everything just "hangs" there, until the task eventually times out.
Any pointers on what I might be missing here, or other workarounds would be very appreciated.
Thank you.
EDIT 1 (2021/02/13):
So I went back to the AWS docs to see if I had missed anything in setting up the VPC endpoints. I originally had one setup for DynamoDB, but since I also need to use S3 in my service, I went ahead and setup a Gateway VPC Endpoint for S3 too (I also wanted to see if the issue I am having is a generic network problem, or specific to DynamoDB).
Then, I made some changes to my script to try to make a call to S3 (to get the bucket's location, for simplicity) as the very first thing to do. I knew that the call would end up timing out, so I wanted to trigger the error immediately upon starting my script execution.
I waited until my task would eventually fail because of the timeout, and this time I noticed something interesting.
Here is the error logs I got when the task failed:
The IP address that my task was trying to reach was 52.85.146.194:443.
And here are the IP addresses that I found in the managed prefix list for S3, which I found in the VPC console:
The IP address I got the timeout error from is not on the list. Could this be a hint to the cause of the issue? Or am I missing something and there is actually nothing wrong with that?
I have a VPC with 2 public subnets in different availability zones (public subnet-2a and public-subnet-2b). I have two private subnets (private-2a-EB-Instance, private-2b-EB-Instance), and two other private subnets for Elasticache (private-2a-EB-Instance, private-2b-EB-Instance), these two subnets for elasticache are part of a subnet group that I use when creating my Elasticache clusters.
I have create a security group for my ElastiCache(redis) and I have changed its inbound rules so it can accept connection from the security group of my ElasticBeanstalk environment. Didn't solve the problem.
I have followed the instructions on AWS' website, and I can't get it to work. I wonder what I am doing wrong. I am using socket-io-redis and I am wondering if that may be part of the problem.
I have used the .config file offered here , but I get an error when deploying the app. It says there was an error creating it.
I created the cluster from the console and tried to connect it to my EB app using the "Primary Endpoint", but I get the error:
Error: Redis connection to https://myelasticache.xxxxxxxxx.cache.amazonaws.com:6379 failed - getaddrinfo ENOTFOUND https://myelasticache.xxxxxxxxx.cache.amazonaws.com
What am I doing wrong? What am I missing?
I think the problem is that you have included https:// in what is supposed to be the hostname. Redis does not connect over HTTP, it uses its own protocol. The error message getaddrinfo ENOTFOUND https://myelasticache.xxxxxxxxx.cache.amazonaws.com indicates that it is trying to resolve https://myelasticache.xxxxxxxxx.cache.amazonaws.com when you want it to use myelasticache.xxxxxxxxx.cache.amazonaws.com.
So try removing https:// and see if it works or if you get another error.
I started a cluster in aws following the guides and then went about following the guestbook. The problem I have is accessing it externally. I set the PublicIP to the ec2 publicIP and then use the ip to access it in the browser with port 8000 as specified in the guide.
Nothing showed. To make sure it was actually the service that wasn't showing anything I then removed the service and set a host port to be 8000. When I went to the ec2 instance IP I could access it correctly. So it seems there is a problem with my setup or something. The one thing I can think of is, I am inside a VPC with an internet gateway. I didn't add any of my json files I used, because they are almost exactly the same as the guestbook example with a few changes to allow my ec2 PublicIP, and a few changes for the VPC.
On AWS you have to use your PRIVATE ip address with Kubernetes' services, since your instance is not aware of its public ip. The NAT-ing on amazon's side is done in such a way that your service will be accessible using this configuration.
Update: please note that the possibility to set the public IP of a service explicitly was removed in the v1 API, so this issue is not relevant anymore.
Please check the following documentation page for workarounds: https://kubernetes.io/docs/user-guide/services/
I have a few EC2 instances.
Only one instance is accessible (ssh port 22) from the Internet.
I want to connect to the others EC2 instances in ssh through the first one.
The public key is set on all EC2 instance.
I enable the forward agent (.ssh/config) for my first instance:
Host my-first-instance
ForwardAgent yes
I thought it was enough but when I connect to the first instance and try to connect to another one (ssh my-second-instance-private-IP), I have a "permission denied". They both use the same user.
I use the same ssh key to access Github from the first instance and it works (so the ForwardAgent works as well).
I don't want to use a VPC feature.
Any idea ?
Thanks in advance.
Jeremy
I had done similar thing earlier by changing the ip-tables of the public instance so that I can connect to different private instances by connecting on different ports. Here is the link to blog that I wrote to explain the same. Hope it helps
http://cloudpages.wordpress.com/2013/08/05/ssh-to-an-instance-in-private-subnet/
Bests,
Himanshu
You are getting permission denied probably because you may not have pub keys of your jump server in your ec2 server's authorized_keys. please do the following -
ssh to the 1st ec2 instance.
generate public and private key pair using keygen.
put the content of the public key (default is .ssh/id_rsa.pub (in case you are
using rsa)) in other ec2 instance's .ssh/authorized_keys.
Now check again.