There are couple of questions around this, but none seem to solve my specific problem.
So i have a publicly accessible RDS instance in a VPC with two Security Groups : one the default SG for VPC & other to allow incoming connections from our developer machines. I am able to successfully connect to this RDS via my developer machines(personal laptops/PCs) as a result of the 2nd Security group.
Now, when i try to connect to this RDS via a Lambda running in same VPC, it fails with this error : "java.net.UnknownHostException: ****.rds.amazonaws.com: Name or service not known "
My lambda is getting launched in the same VPC as RDS and i have chosen all the RDS VPC subnets for the lambda. It also is tied to the default VPC security group.
Below is my understanding(obviously not right as connection is not working), what am i missing here?
The Default VPC SG(security group) associated to the lambda has outgoing rules set as 0.0.0.0/0 , hence it should allow lambda to access the whole world; so at least the RDS in VPC should be accessible.
The same default VPC SG should allow the RDS to accept incoming connections from any ip in the SG subnets(inbound rule mentions SG/default) ; hence lambda should be able to access it.
The error message is indicating that it cannot resolve the DNS Name of the RDS database (Name or service not known).
I suspect that this is due to the fact that the RDS instance has been configured as publicly accessible, which gives it a public DNS Name and IP address.
An AWS Lambda function connected to a VPC will send all network traffic via the VPC. It does not have access to the Internet unless there is a NAT in the VPC and the Lambda function is connected to a Private Subnet. Thus, the connection attempts from the Lambda function are possibly attempting to resolve a Public DNS Name, but is failing to do so.
This could likely be resolved by changing the RDS instance to be Publicly Available = No. This not only makes it safer, but it means that the DNS Name will resolve to a Private IP address, which should work correctly with a Lambda function connected to the VPC.
Alternatively, you could configure the Lambda function to not be connected to the VPC. This gives it Internet access, so it will connect to the RDS database just like any other computer on the Internet.
So the problem was that in my developer machine, I was using the db instance endpoint , but in lambda i was using the cluster endpoint. They look similar, hence i was not able to catch it earlier. On fixing the lambda to use the instance endpoint, both lambda & developer machines can successfully connect to the RDS.
However, I am now curious why the cluster url is not accessible, but i will create another question for that.
Related
I have an RDS in one AWS Account - say Acct-1.
The RDS is public (i know it's not a good idea and there are other solutions for that)
I have a lambda in another AWS Account - say Acct-2 which runs in a VPC.
I have setup VPC peering between the 2 accounts, the route table entries are in place as well as the security groups IN/OUT bound policies in place.
In Acct-2 I can verify that I can connect to the RDS instance in Acct-1 using a mysql cient from an EC2 instance. The EC2 instance is in the same subnet as the Lambda and they both have the same security group.
But the Lambda gets a timeout connection. The Lambda has the typical Lambda execution role that Allows logs, and network interfaces.
Thoughts on what could be missing ? Does the RDS need to grant specific access to the Lambda service even if it's running in a VPC ?
Clarification: There is no route to the RDS instance from the internet. Clearly, the ec2 host is able to resolve the Private IP for the RDS instance from the DNS name and connect.
Lambda is unable to resolve the private IP for the RDS instance.
I'm trying to keep the traffic within AWS so as to not pay egress costs.
I've created an EC2 instance inside a public subnet (so that I can access it from my home network) and I have created some Lambda's inside the private subnets of my VPC.
My 1st lambda can freely access the internet (through a NAT Gateway) and do its job. Which is to fetch a file from the internet and upload it to S3, once per day.
My 2nd lambda is supposed to retrieve this file from S3 (which it does without issue) read the file & then upload the data to MySQL running on the EC2 instance. It is unable to connect to the Database (using either the EC2's public or private IP's) and the Cloudwatch logs show that the session times out, making me think this is a networking issue.
I have a 3rd lambda that will also need to interact with the EC2/DB instance.
My security group allows for all incoming traffic from my home network IP, the NAT gateway & the VPC IP range. All outbound traffic is allowed.
I appreciate its not usual to have an EC2/DB set up this way, in a public subnet, but its my preference to interact it with it this way from home using Sequel Pro.
However, is my set up even possible? Eg can my private subnet lambdas interact with a public subnet ec2 instance? if so, does anybody have any ideas how I can make this happen?
It appears that your situation is:
An Amazon EC2 instance running in a public subnet, with MySQL
The EC2 instance has a Security Group allowing all incoming traffic from your home network IP, the NAT gateway and the VPC IP range
An AWS Lambda function connected to a private subnet of the same VPC
A NAT Gateway allowing private subnets to connect to the Internet
The Lambda function is unable to connect with the MySQL database running on the EC2 instance
The normal security configuration for this scenario would be:
A Security Group on the Lambda function (Lambda-SG) that allows all Outbound access (no Inbound required)
A Security Group on the EC2 instance (EC2-SG) that allows inbound access from Lambda-SG on port 3306, plus whatever inbound permissions you want for accessing your instance via SSH, etc.
Given that your Security Group includes "the VPC IP range", this should be sufficient to permit inbound access.
The Lambda function should reference the EC2 instance via its private IP address to keep traffic within the VPC. By default, all subnets within a VPC can communicate with each other unless the Network ACLs have been modified (and they should generally be left at default values).
This means that the only remaining explanation would be that the MySQL database is not accepting traffic from the VPC IP range. (I'm not a MySQL person, but I know that PostgreSQL requires incoming IP ranges to be defined, so this might be true for MySQL too.)
To diagnose what might be happening in your network, I recommend:
Launch another Amazon EC2 instance in the public subnet
Connect to the new instance and try to connect to the MySQL database via the private IP address
If that works, repeat the process but from an EC2 instance in the private subnet. To use this you will need to connect to the 'public' EC2 instance, and from there connect to the 'private' EC2 instance. Then, try and connect to MySQL from that private instance.
These steps will progressively let you identify where the network problem might lie. Let us know what you find!
So I have a very confusing issue that I don't know how to solve. My setup is API Gateway -> Lambda -> IoT Core. I setup the code and it works fine from my IDE. I deploy it to AWS, and my connection to AWS times out.
The Lambda is in a single subnet and the subnet does have a default route to an IGW. I did a test, and the Lambda function can resolve the IP of my IoT endpoint to a public IP (54.x.x.x). But the connect() method times out. My security group for the Lambda function is setup to allow all incoming / outgoing.
What am I missing? Why can't I get to IoT Core from inside a VPC with an IGW configured and seems to be working. Any direction would be greatly appreciated.
UPDATE
After playing around with many different things, I can't identify what exactly I had messed up with my configuration. But following the accepted answer I ended up with the following setup which appears to work for what I need.
subnet-1 10.14.10.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-2 10.14.20.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-3 10.14.30.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-4 10.14.40.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-5 10.14.200.0/24 (auto-assign-public=true)
local route ( 10.14.0.0/16 ) and default route=igw
nat-gateway
in subnet-5
I don't know if this is what I intended, but this is what I was looking for. A series of subnets that are not publicly accessible, but has an internet connection for access to other AWS services. So my Lambda resources, ECS, etc can sit privately and access what they need.
Thank you everyone for the information.
You should not deploy the Lambda function to a public subnet (that's the subnet with the default route to the IGW). It won't work the way you want it to work. The Lambda function doesn't have, and cannot have, a public IP so cannot route to the internet via the IGW.
If the Lambda needs to be in VPC, then move it to a private subnet and make sure that the private subnet has a default route to a NAT (or NAT gateway) in a public subnet. Or deploy the Lambda function outside of VPC completely, if that's viable.
More information at:
Why can't Lambda in public subnet reach the internet?
Why do we need private subnet in a VPC?
When you say "I did a test, and the Lambda function can resolve the IP of my IoT endpoint to a public IP (54.x.x.x)" Do you mean DNS resolution, or you've checked this with a actual network traffic.
In either case, you can turn VPC Flow Logs for your VPC, and try again. The flow log will identify whether SGs or NACLs are blocking your traffic.
Remember also that Lambda's cannot exists in a public subnet, they have to reside in private subnets, and use NAT GW on public subnets to connect to the internet.
I encountered the same issue. Thankfully, AWS has automated this process and it's only a few clicks but sadly an extra $$$/month (about $30 to start).
You need to create a VPC group that has both public and private subnets. For my case, I was accessing a RDS database to then build email templates and fire off these emails via SES.
What did not work:
When I deployed a default VPC on my lambda function, the RDS would work, but the SES would not. The reason being is the SES API has no access to the internet via default VPCs, you need to setup a private subnet to allow for this.
VPC wizard can do this all for you with a few simple clicks, but you're looking at spending about $30/month (CAD) to setup a NAT gateway + any potential data processing charges per GB (read the pricing). For my case, firing a few email templates off it outweighed deploying another ec2 instance and gave me flexibility for scaling without too much on-the-fly configuration.
This link is a good place to get started. How do I give internet access to a Lambda function that's connected to an Amazon VPC?
Follow the VPC wizard hyperlink within that article, and you just need to go to the VPC console and create a VPC, should look something like this (below)
Once AWS works its magic, you need to assign the VPC to your Lambda function, assign the right permissions to your lambda function AND don't forget to configure your security group with the correct inbound rules. In my case I need to add a PostGres rule to access my RDS.
Hope this helps!
I was running a serverless web application on a lambda inside a VPC, and connecting to a Aurora-MySQL RDS instance, with inbound rules to allow traffic from the security group of the lambda
The connection was working fine, however, quite often the lambda cold start was giving me a timeout.
After some research, I found out that running a lambda on a VPC brings an additional cost on startup and I saw the recommendation in more than 1 place to avoid using lambda on a VPC except if you strictly need to access some resource in the VPC.
So, I decided to move my RDS to a publicly accessible instance, so my lambda can access it over the internet and remove the lambda from the VPC.
So, I changed the RDS Public accessibility option to Yes and edited the security group to allow inbound connection from any IP.
I have also removed the VPC from the lambda, so the lambda is not running on a VPC anymore
I thought it was gonna be enough.
But then my lambda started failing to connect to the database
I tried to connect using my local client, again, failure
tried pinging to the hostname, got request timeouts
After digging a bit into it, I found that my DB instance subnet group having some private subnets might be a problem (?)
So, I have created a new subnet group with only public subnets, and tried to move my db instance to the new subnet group... but got this message:
You cannot move DB instance my-instance to subnet group my-new-group. The specified DB subnet group and DB instance are in the same VPC.
Ok, it seems that I can't move to a different subnet in the same VPC, I started trying to create a new VPC, but it doesn't seem to be right and I'm sure there is something else I am missing here.
I also read about Network ACL, and thought that this might be the problem, but my rules seem to be fine, with the default rule to allow any traffic (and the rule * to DENY)
ALL Traffic ALL ALL 0.0.0.0/0 ALLOW
My RDS Network settings
Subnet group
default
Subnets
subnet-11111111
subnet-22222222
subnet-33333333
subnet-44444444
subnet-55555555
subnet-66666666
Security
VPC security groups
default (sg-111111)
( active )
Public accessibility
Yes
My Security group inbound rules
Type Protocol Port range Source Description - optional
All traffic All All 0.0.0.0/0 -
All traffic All All ::/0 -
Still can't connect, can't connect with my local client, can't even ping it:
Connecting through my local client
Can't connect to MySQL server on 'my-instance.xxxxxxxxxx.us-east-1.rds.amazonaws.com'
ping my-instance.xxxxxxx.us-east-1.rds.amazonaws.com
PING ec2-xx-xx-xx-xx.compute-1.amazonaws.com (xx.xx.xx.xx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Any idea of what I am missing here?
UPDATE
My VPC has internet access (I can access internet services from it, not an issue), I have an Internet Gateway and NAT Gateway in place.
I'm using Zappa for the lambda deployment, which takes care of creating a keep-warm function... however, I know that concurrent requests could still be an issue
The issue with VPC in lambda is that it can add 10s on the cold start, which is a no-deal for some of my use cases:
https://www.freecodecamp.org/news/lambda-vpc-cold-starts-a-latency-killer-5408323278dd/
Besides enabling "public access" on your RDS instance, you need to enable internet to the VPC using an internet gateway. After you attach it to the VPC, you need to route the data from the database subnets to the internet gateway. Check here
But I would not advise you expose your database like this. If you are having issues with lambda cold start, you should create an event to keep it warm.
Things you need to is :
Create new subnet group with default VPC
assign two subnet for availability zone
then modify your RDS instance
change subnet group to newly created group
mark "Publicly accessibility" to Yes.
Check your VPC is using internet gateway or not.
Check lambda security group whether it's open for outbound connection for Database port is not available or not.
No need to create different VPC for RDS. Use Default VPC.
As recommended by #stargazer try to not to expose publicly or out of VPC. Its works well inside VPC.
How can I create this scenario:
- A private subnet-1
- Lambda in subnet-1
- RDS in subnet-1(same subnet with Lambda)
- Both inside the same VPC.
is the above feasible and is it a good architecture ?
I have tried implementing it only issue is, it's timing out. But, when the Lambda is out of the subnet-1 it's working fine.
My understanding is since both RDS and Lambda are in the same subnet they should easily communicate.
It is perfectly okay to have Amazon RDS and the AWS Lambda function connected to the same private subnet.
Some things to note:
If the Lambda function also requires access to the Internet (eg to make calls to Amazon S3), then the VPC will also require a NAT Gateway in a public subnet.
The Lambda function should refer to the RDS instance by DNS Name. This should resolve to a private IP address local to the VPC.
The Lambda function should be assigned a Security Group (eg Lambda-SG)
The security group associated with the RDS instance (RDS-SG) should permit inbound access on the appropriate port (eg 3306 for MySQL) from Lambda-SG.
That is, RDS-SG should permit inbound connections from Lambda-SG.