This question has been asked several times, and I've been reading the answer for the past hours, so I'm going to summarise everything here.
To get the connection between RDS and lambda, the normal steps are:
Attach the lambda to the same VPC as the database
Create a security group for the lambda
Modify / add a security group for the RDS, in which you allow the VPC's security group as inbound providing the connection port.
Add policy AWSLambdaVPCAccessExecutionRole to lambda function
That's supposed to be it, however it is not working for me.
I have checked other possible errors:
DNS of database not found. Solved by setting an internet gateway for the VPC, checked that the DNS is properly resolved
Inbound and Outbound rules of the lambda functions. Solved by setting literally all traffic for both directions.
Outbound rule for DB's security group is set to all traffic and destinations
Note: I can connect to the database via a VPN using another security group, meaning it's not a credential error.
I really don't understand what else I'm missing at this point
Finally found the answer, the error was due to the fact that the RDS was on a public subnet which had an internet gateway. In this case, the solution is as follows:
Create a public NAT gateway (attached to a public subnet, with an elastic ip)
Create one (or more) private subnets in the same VPC
Create a route with the “Destination” as “0.0.0.0”, “Target” as the Internet Gateway, and save your settings. Associate this route with the public subnets
Create a route in this table with the “Destination” as “0.0.0.0”, “Target” as the NAT Gateway created previously, and save your settings. Associate this route with the private subnets
Finally, connect the lambda to the private subnet(s) only
Related
I'm trying to figure out the following:
I have an API service deployed in the default VPC, and I have a ELB configured to access the service.
The ELB is attached to a security group sg-XXXXXXXX1 to restrict inbound traffic (open to all for outbound)
I'm now trying to create a lambda function that can call the API service. (using python requests)
I've tried the following and failed to succeed:
In the configuration > VPC section, I added sg-XXXXXXXX1 as the security group and the 4 default subnets
Added AWSLambdaVPCAccessExecutionRole to lambda role
Created a new security group sg-XXXXXXXX2 (all traffic for inbound and outbound), attached it to lambda, and added sg-XXXXXXXX2 to the inbound list (all traffic) for sg-XXXXXXXX1
Any ideas on what I did wrong? and how I can fix it?
Your ELB is most likely a public ELB (which is the default setting). The ELB DNS will resolve to a public IP address, not a VPC IP address. That means that the traffic going to the ELB will exit the VPC, go out to the Internet, and then back into AWS and into the ELB. When that happens, any association with the Lambda function's security group gets lost. Also, if your Lambda function isn't running in a subnet with a route to a NAT gateway it won't be able to access the Internet at all, so it is just going to timeout and fail when it tries to access the ELB.
To fix this, change the load balancer scheme to private, which will give the load balancer a private VPC IP address, and make it accessible only within the VPC.
I've spend about a full day trying to solve this, but have no luck so far. I'm also open to alternative suggestions than my current setup.
I have an RDS instance inside of a VPC. I am trying to make CodeBuild be able to access this RDS instance for a testing step.
Currently, I setup a VPC endpoint for the CodeBuild service, with all 3 subnets of the VPC. I know that if I allow all inbound traffic for the security group on the RDS, it works. I don't want to allow all inbound traffic though- and given this, have been unsuccessful.
I have tried the following to no avail:
Taking the private IPv4's of the ENI's created by the VPCE, adding them as inbound rules to the security group on the RDS
Creating a separate VPC for CodeBuild, and setup VPC peering (this seemed overly complex, and I'm not sure if the peering would even allow CodeBuild traffic to hit an RDS; it also makes things complicated down the road for CodeDeploy).
Putting CodeBuild inside the VPC of the RDS instance. When doing this, I created a new subnet in the VPC, assigned it to a NAT in the routes table (and this NAT was on the VPC of the RDS instance); put CodeBuild kept telling me it had no internet access.
setup a VPC endpoint for the CodeBuild service,
VPC endpoints are not used for inbound traffic from CB to VPC. They are used for your applications in VPC to interact with CB service without the internet.
Putting CodeBuild inside the VPC of the RDS instance.
This is the correct way. Sadly you haven't provided any details of your VPC, subents, NAT, route tables, security groups, NACLs setup, thus its difficult to speculate why it does not work.
Thanks Marcin for pointing me in the right direction to make CodeBuild in the same VPC. When I was able to focus on that, I saw this post again:
CodeBuild cannot find the 0.0.0.0/0 destination for the target internet gateway
which I had the same issue; my NAT was also on the private subnet. Now, it's on the public subnet, and it's working.
I was running a serverless web application on a lambda inside a VPC, and connecting to a Aurora-MySQL RDS instance, with inbound rules to allow traffic from the security group of the lambda
The connection was working fine, however, quite often the lambda cold start was giving me a timeout.
After some research, I found out that running a lambda on a VPC brings an additional cost on startup and I saw the recommendation in more than 1 place to avoid using lambda on a VPC except if you strictly need to access some resource in the VPC.
So, I decided to move my RDS to a publicly accessible instance, so my lambda can access it over the internet and remove the lambda from the VPC.
So, I changed the RDS Public accessibility option to Yes and edited the security group to allow inbound connection from any IP.
I have also removed the VPC from the lambda, so the lambda is not running on a VPC anymore
I thought it was gonna be enough.
But then my lambda started failing to connect to the database
I tried to connect using my local client, again, failure
tried pinging to the hostname, got request timeouts
After digging a bit into it, I found that my DB instance subnet group having some private subnets might be a problem (?)
So, I have created a new subnet group with only public subnets, and tried to move my db instance to the new subnet group... but got this message:
You cannot move DB instance my-instance to subnet group my-new-group. The specified DB subnet group and DB instance are in the same VPC.
Ok, it seems that I can't move to a different subnet in the same VPC, I started trying to create a new VPC, but it doesn't seem to be right and I'm sure there is something else I am missing here.
I also read about Network ACL, and thought that this might be the problem, but my rules seem to be fine, with the default rule to allow any traffic (and the rule * to DENY)
ALL Traffic ALL ALL 0.0.0.0/0 ALLOW
My RDS Network settings
Subnet group
default
Subnets
subnet-11111111
subnet-22222222
subnet-33333333
subnet-44444444
subnet-55555555
subnet-66666666
Security
VPC security groups
default (sg-111111)
( active )
Public accessibility
Yes
My Security group inbound rules
Type Protocol Port range Source Description - optional
All traffic All All 0.0.0.0/0 -
All traffic All All ::/0 -
Still can't connect, can't connect with my local client, can't even ping it:
Connecting through my local client
Can't connect to MySQL server on 'my-instance.xxxxxxxxxx.us-east-1.rds.amazonaws.com'
ping my-instance.xxxxxxx.us-east-1.rds.amazonaws.com
PING ec2-xx-xx-xx-xx.compute-1.amazonaws.com (xx.xx.xx.xx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Any idea of what I am missing here?
UPDATE
My VPC has internet access (I can access internet services from it, not an issue), I have an Internet Gateway and NAT Gateway in place.
I'm using Zappa for the lambda deployment, which takes care of creating a keep-warm function... however, I know that concurrent requests could still be an issue
The issue with VPC in lambda is that it can add 10s on the cold start, which is a no-deal for some of my use cases:
https://www.freecodecamp.org/news/lambda-vpc-cold-starts-a-latency-killer-5408323278dd/
Besides enabling "public access" on your RDS instance, you need to enable internet to the VPC using an internet gateway. After you attach it to the VPC, you need to route the data from the database subnets to the internet gateway. Check here
But I would not advise you expose your database like this. If you are having issues with lambda cold start, you should create an event to keep it warm.
Things you need to is :
Create new subnet group with default VPC
assign two subnet for availability zone
then modify your RDS instance
change subnet group to newly created group
mark "Publicly accessibility" to Yes.
Check your VPC is using internet gateway or not.
Check lambda security group whether it's open for outbound connection for Database port is not available or not.
No need to create different VPC for RDS. Use Default VPC.
As recommended by #stargazer try to not to expose publicly or out of VPC. Its works well inside VPC.
I have a lambda function that makes a REST call. While developing, I had tagged it to "No VPC" and it was working. Since, the function needed to talk to RDS, I had to tag to VPC that hosted RDS. Since then REST call is not working.
I followed the article -
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
The VPC is tagged to default security group. The security group has both inbound and outbound allowed
The VPC has 6 subnets. 5 are named as private subnet and 1 is named as public subnet
The public subnet is linked to public router which is linked to internet gateway
All the 5 subnets are linked to private router that is linked to NAT Gateway
ACLs have both inbound and outbound allowed
The lambda function was linked to two private subnets
When I try my lambda function, it still doesnt work.
Further analysis revealed that NAT gateway is linked to ENI-XXXXX through elastic IP. When I check the inbound and outbound rule for ENI-XXX, there is nothing listed. When I tried to change the security group, I received - You do not have permission to access the specified resource. Please note, I was trying to change it through root account
I tried creating a Elastic IP, and tagged it to existing eni-xxx that has inbound and outbound rule. However I am not able to link it to new NAT(the new NAT fails with message that EIP is already associated). If I again create a new NAT Gateway, it creates a new ENI-XXXX which has no inbound or outbound and I dont find option of linking new NAT to an ENI that has inbound and outbound rule.
Has anyone faced this issue? Please let me know where am I going wrong?
I was struggled this too and resolved by this:
Create a NAT gateway into the public subnet, then it will be linked to public IP.
Now, go to the Routing table in the VPC service and create it.
Linked subnets should be the private subnets of lambda.
destination to your VPC as local
destination to 0.0.0.0/0 as NAT gateway what you created a bit ago.
Test your lambda.
I didn't touch any ENI things.
I am trying to setup a nice and secure VPC for my lambda and RDS work. Essentially, I want my lambda to hit a site, get some data, and shove it into a database.
In isolation the parts all work. However the second I go to harden everything it all falls apart. Here is what I do:
Disable "Publically Available" from the RDS instance
Change the RDS instance to only accept connections from inside the VPC using the security group
Associate the lambda with a VPC (this kills the internet access)
Following this tutorial I created a NAT gateway, deleted the internet gateway from the VPC subnet, and replaced it with the NAT. Now, as expected, nothing can talk inbound, but things can talk outbound.
At this point I knew I needed a bastion instance, so I fired up an EC2 instance.
The EC2 instance is set to the same subnet the RDS and Lambda are on, and unfortunately this means that I have a problem - the NAT gateway is currently soaking up all the traffic via 0.0.0.0/0, which means there's no room for the internet gateway. Without the internet gateway I (obviously) can't SSH into my bastion instance so I can jump to access my RDS database.
How can I configure this all correctly? My guess is that I need to split the subnet up somehow and make a private and public subnet, the public having the bastion and internet gateway in it. However, I'm not sure how this will all work so the bastion instance can still properly jump to the RDS.
I'm really quite new to setting up AWS services so I'm hoping I didn't mess anything up long the way.
Following this tutorial I created a NAT gateway, deleted the internet
gateway from the VPC subnet, and replaced it with the NAT. Now, as
expected, nothing can talk inbound, but things can talk outbound.
Short Answer
The short answer is you shouldn't have "Killed the Internet Gateway"; thats not a step in the link you provided :) Leave the internet gateway as is in your current subnet. You're going to need a public subnet and the one that was routing 0.0.0.0/0 to IGW is an example of one you can could use.
The work involved is placing your NAT gateway in the Public Subnet, placing your bastion host in the public subnet, placing your lambda function in the private subnet, routing traffic in the private subnet to the NAT gateway in the public subnet, and providing your lambda function with access to your security group by putting it in its own lambda security group and "white listing" the lambda security group in the inbound rules for the security group protecting your database.
Background
Below I have an expanded answer providing background as to public/private subnets, granting internet access to private subnets, and allowing lambda access through security groups. If you don't feel like reading the background then jump to very end where I give a bullet point summary of the steps you'll need.
Public Subnet
A public subnet is one in which traffic originating outside your VPC, or destined for a target outside your VPC (internet), is routed through an internet gateway (IGW). AWS gives you initial default public subnets configured this way; you can identify them in the console by looking at their route table and seeing that under "destination" you find "0.0.0.0/0" targeting an IGW. This means a public subnet is more of a design pattern for "internet accessible" subnet made possible by simply configuring its default route to point to an IGW. If you wish to create a new public subnet you can create a new route table as well that point internet traffic at an IGW and link that route table to your new subnet. This is fairly easy in the console.
Private Subnet
A private subnet is a subnet with no IGW and not directly reachable from the internet, meaning you cannot connect to a public IP address of a system on a private subnet. With the exception of the AWS pre-configured default subnets, this is how new subnets your manually create are setup, as black boxes till you specify otherwise.
Granting Internet Access to Private Subnet
When you want things in your private subnet to be able to reach out to external internet services you can do this by using an intermediary known as a NAT gateway. Configure a route table the same as in the public subnet with the only difference being traffic destined for 0.0.0.0/0(Internet) you target for a NAT gateway sitting inside the public subnet. That last part is critical. Your NAT gateway needs to be in the public subnet but your private subnet is using it as the target for external traffic.
Security Group Access for Lambda
One simple way to allow your lambda function through your security group/firewall is to create a security group just for your lambda function and configure the security group protecting your RDS so that it allows traffic from the lambda security group.
In other words, in security group settings you don't have to specify just IP addresses as sources, you can specify other security groups and this is a pretty neat way of grouping items without having to know their IP address. Your lambda functions can run in the "Lambda Security Group" and anything protected by a security group that you want them to access can be configured to accept traffic from the "Lambda Security Group". Just make sure you actually associate your lambda function with the lambda security group as well as place it in the private subnet.
Lambda VPC Steps in a Nutshetll
Create a new NAT gateway and place it in the public subnet. This
point is important, the NAT gateway goes in the public subnet ( a
subnet whose route table routes 0.0.0.0/0 to an IGW)
Create a new subnet, you can call it Private-Lambda-Subnet. Unlike
the
default pre-configured subnets AWS gives you, this new subnet is
immediately private out of the box.
Create a new route table and link it to your Private-Lambda-Subnet
In the new route table for your private subnet add an entry that
routes 0.0.0.0/0 to a target of the NAT gateway. This is how your
private subnet will indirectly access the internet, by forwarding
traffic to the NAT which will then forward it to the IGW.
Your bastion host and anything else you want to be be publicly
accessible will need to be in the public subnet. This is probably
where you already have your RDS instances, which is fine if they are
firewalled/security group protected.
Create a new security group for your lambda function(s). You can
call it LambdaSecurityGroup.
Configure the inbound rules of your RDS guarding security group to
allow traffic from the LambdaSecurityGroup. This is possible because
you can use other security groups as sources in the firewall
settings, not just ip addresses.
You need a public subnet (default route is the Internet Gateway) and a private subnet (default route is the NAT Gateway). The NAT Gateway, itself, goes on the public subnet, so that it can access the Internet on behalf of the other subnets for which it is providing services. The bastion also goes on the public subnet, but Lambda and RDS go on the private subnet.
Anything can talk to anything on any subnet within a VPC as long as security groups allow it (and Network ACLs, but don't change these unless you have a specific reason to -- if you aren't sure, then the default settings are sufficient).