I am trying to access Timestream from EC2/Lambda instances that run within a VPC so that I can speak to a RDS instance from those EC2 instances/Lambda functions. I have spent many hours trying to get access to Timestream via PrivateLink/a VPC instance endpoint to work and think I may have found an issue. When I provision a VPC endpoint for the Timestream ingest service, the Private DNS name is specific to the cell endpoint, e.g. ingest-cell2.timestream.us-east-1.amazonaws.com NOT the general endpoint URL that boto3 uses, i.e. ingest.timestream.us-east-1.com. When I run a nslookup on ingest-cell2.timestream.us-east-1.amazonaws.com it properly resolves to the private IP of the VPC endpoint ENI, but if I lookup the more general endpoint URL of ingest.timestream.us-east-1.com it continues to resolve to public AWS IPs. The result of this is that if I initialize the timestream write client normally and perform any actions, it hangs because it is trying to communicate with a public IP from a private subnet,
import boto3
ts = boto3.client('timestream-write')
ts.meta.endpoint_url # https://ingest.timestream.us-east-1.amazonaws.com
ts.describe_endpoints() # hangs
ts.describe_database(DatabaseName='dbName') # hangs
If I explicitly give it the cell specific endpoint URL, the describe_endpoints() function throws an error but seemingly normal functions work (haven't tested writes or reads yet, just describing databses)
import boto3
ts = boto3.client('timestream-write', endpoint_url='https://ingest-cell2.timestream.us-east-1.amazonaws.com')
ts.describe_endpoints() # throws UnknwonOperationException error
ts.describe_databse(DatabaseName='dbName') # Succeeds
If I provision a NAT gateway in the private subnet rather than a VPC endpoint everything works normally as expected. Furthermore for fun, I tried adding the VPC endpoint private IP to the /etc/hosts file with ingest.timestream.us-east-1.com to force proper resolution and even then I get the same hanging behavior when running the above block of code
This seems pretty broken to me. The whole point of the VPC endpoint is to enable the SDK to operate normally. Maybe I am missing something?
Related
For example, in my python code inside lambda I have the following string:
conexao = (
"Driver={/opt/microsoft/msodbcsql18/lib64/libmsodbcsql-18.1.so.2.1};"
"Server=example.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com;"
"Database=example;"
"TrustServerCertificate=yes;"
"Uid=admin;"
"Pwd=xxxxxx;"
)
The endpoint of my RDS database is example.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com
Both RDS database and lambda database is in the same VPC and subnet.
Since it's in the same VPC and subnet, this mean I am not going over the internet right? or maybe it is? my RDS has the option Publicly accessible = Yes (because I need to access through my SSMS).
I'd like to access over the internet only in SSMS, when accessing thought lambda I'd like to use privatly.
Does AWS understand that, in this case, python code inside lambda don't need to access over the internet?
this mean I am not going over the internet right?
Yes, that's correct.
You can very easily check it yourself if you want. In your lambda function, resolve example.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com into an IP address. What you should get, is a private IP address of the RDS instance, not public.
So, let me start in reverse. The issue is that there is no issue. The seemingly relevant Google docs state (https://cloud.google.com/sql/docs/postgres/connect-functions):
If you route all egress traffic through the VPC connector, you
must [my use of bold font] use a Private IP.
Let me now describe my set-up.
1 Google function using a VPC connector. It's invoked via Pub/Sub.
Ingress configuration: internal only;
egress configuration: route all traffic through VPC.
Actions taken when the function is called:
connect to an external API using a whitelisted static IP (provided by virtue of using the aforementioned VPC)
connect to a DB using its internal IP (the same virtual network)
connect to another DB using its connection name (I assume an SQL Proxy tunnel is used implicitly to achieve that)
All work alright. Yet if I understand the docs correctly the third action is bound to fail.
P.S. The context behind this manipulation is: I want to connect to 2 DBs, but as far as I understand that is impossible to achieve with VPC peering (or any current Google solution) if both are in different virtual networks and have only private IPs. I saw multiple years-old feature requests.
I have an EC2 instance running inside an ECS cluster with VPC.
On the instance, I need to run a ECS task that needs access to DynamoDB.
When I try running the same task using Fargate, I can use the assignPublicIp = 'ENABLED' option to allow my task to have access to other AWS services, and everything works fine.
However, the assignPublicIp option is not available for the EC2 launch type, and I cannot figure out how to allow my EC2 instance have access to other AWS services.
I read the AWS docs and followed guides like this one to setup VPC endpoint for DynamoDB.
I also made sure that there aren't any network access restrictions by making sure that inbound/outbound rules for my NACL and security group for the VPC are wide open (at least for the sake of testing).
Here is how the rules look like, for both NACL and my security group:
Finally, I used the VPC > Reachability Analyzer to check if AWS can detect any problems regarding the connection path between my EC2 instance and DynamoDB, but the analysis reported a Reachable status.
It basically told me that there was no issues regarding establishing a connection along the following path:
Network interface for my EC2 instance (source)
Security group for the VPC
NACL for the VPC
Route table for the VPC
which includes the following route added by the VPC endpoint for DynamoDB
Destination: pl-02cd2c6b (com.amazonaws.us-east-1.dynamodb, 3.218.182.0/24, 3.218.180.0/23, 52.94.0.0/22, 52.119.224.0/20)
Target: the endpoint ID (e.g., vpce-foobar)
VPC endpoint for DynamoDB (destination)
Despite AWS telling me that I have a "Reachable" status, I still think it might be a network reachability problem, because when I run the task, the script I am running gets stuck right after it makes a GetItem call to DynamoDB.
If it was a permission error or an invalid parameter issue, I would get an error immediately, but everything just "hangs" there, until the task eventually times out.
Any pointers on what I might be missing here, or other workarounds would be very appreciated.
Thank you.
EDIT 1 (2021/02/13):
So I went back to the AWS docs to see if I had missed anything in setting up the VPC endpoints. I originally had one setup for DynamoDB, but since I also need to use S3 in my service, I went ahead and setup a Gateway VPC Endpoint for S3 too (I also wanted to see if the issue I am having is a generic network problem, or specific to DynamoDB).
Then, I made some changes to my script to try to make a call to S3 (to get the bucket's location, for simplicity) as the very first thing to do. I knew that the call would end up timing out, so I wanted to trigger the error immediately upon starting my script execution.
I waited until my task would eventually fail because of the timeout, and this time I noticed something interesting.
Here is the error logs I got when the task failed:
The IP address that my task was trying to reach was 52.85.146.194:443.
And here are the IP addresses that I found in the managed prefix list for S3, which I found in the VPC console:
The IP address I got the timeout error from is not on the list. Could this be a hint to the cause of the issue? Or am I missing something and there is actually nothing wrong with that?
I am new to AWS and I am trying to deploy simple app to AWS ECS. I have two simple docker containers, running in ECS Fargate:
‘Frontend’: Vue Js app, which makes a single request to backend;
‘Backend’: Django app, which serves the request;
Both services were launched within the same cluster, in default VPC and the same, single public subnet. For ‘Backend’ I configured Service Discovery: Namespace – test, Service Discovery Name – backend. Security group configured to allow All Traffic.
So, the problem is when frontend makes request:
axios.get('http://backend.test:8000/api/get-test/')
I got error: Failed to load resource: net::ERR_NAME_NOT_RESOLVED backend.test:8000/api/get-test/
However, executing in AWS Cloud9 command: dig +short backend.test returns correct private IP of the backend container.
When I change request to something like
axios.get('http://172.17.3.85:8000/api/get-test/')
where 172.17.3.85 is valid private IP of the backend container, I got following error:
GET http://172.17.3.85:8000/api/get-test/ net::ERR_CONNECTION_TIMED_OUT
However, if I spin out EC2 instance in the same VPC and subnet and SSH to it, I can ping backend container, and requests -
curl -v http://172.17.3.85:8000/api/get-test/
as well as
curl -v http://backend.test:8000/api/get-test/
return desired response.
The only case when everything is working as expected is when the request is like
axios.get('http://3.18.59.133:8000/api/get-test/'),
where 3.18.59.133 is valid Public IP of the backend container.
I would appreciate any suggestion where look further or how to connect two containers via service discovery as right now I am out of ideas.
Based on the discussion in comments and description of the problem, the reason is that the Frontend’: Vue Js app executes on the client side, for example, in the browser.
This explains all the issues described and discussed:
axios.get('http://backend.test:8000/api/get-test/') does not work as on the client side you can't resolve privte hosted zone.
axios.get('http://172.17.3.85:8000/api/get-test/') does not work because the 172.17.3.85 is valid only in the VPC, not on the client's network.
spin out EC2 instance in the same VPC and subnet and SSH works because private hosted zones can be resolved inside VPC.
axios.get('http://3.18.59.133:8000/api/get-test/') works because public IP can be used on the clinet side, unlike private IPs.
I started a cluster in aws following the guides and then went about following the guestbook. The problem I have is accessing it externally. I set the PublicIP to the ec2 publicIP and then use the ip to access it in the browser with port 8000 as specified in the guide.
Nothing showed. To make sure it was actually the service that wasn't showing anything I then removed the service and set a host port to be 8000. When I went to the ec2 instance IP I could access it correctly. So it seems there is a problem with my setup or something. The one thing I can think of is, I am inside a VPC with an internet gateway. I didn't add any of my json files I used, because they are almost exactly the same as the guestbook example with a few changes to allow my ec2 PublicIP, and a few changes for the VPC.
On AWS you have to use your PRIVATE ip address with Kubernetes' services, since your instance is not aware of its public ip. The NAT-ing on amazon's side is done in such a way that your service will be accessible using this configuration.
Update: please note that the possibility to set the public IP of a service explicitly was removed in the v1 API, so this issue is not relevant anymore.
Please check the following documentation page for workarounds: https://kubernetes.io/docs/user-guide/services/