I was running a serverless web application on a lambda inside a VPC, and connecting to a Aurora-MySQL RDS instance, with inbound rules to allow traffic from the security group of the lambda
The connection was working fine, however, quite often the lambda cold start was giving me a timeout.
After some research, I found out that running a lambda on a VPC brings an additional cost on startup and I saw the recommendation in more than 1 place to avoid using lambda on a VPC except if you strictly need to access some resource in the VPC.
So, I decided to move my RDS to a publicly accessible instance, so my lambda can access it over the internet and remove the lambda from the VPC.
So, I changed the RDS Public accessibility option to Yes and edited the security group to allow inbound connection from any IP.
I have also removed the VPC from the lambda, so the lambda is not running on a VPC anymore
I thought it was gonna be enough.
But then my lambda started failing to connect to the database
I tried to connect using my local client, again, failure
tried pinging to the hostname, got request timeouts
After digging a bit into it, I found that my DB instance subnet group having some private subnets might be a problem (?)
So, I have created a new subnet group with only public subnets, and tried to move my db instance to the new subnet group... but got this message:
You cannot move DB instance my-instance to subnet group my-new-group. The specified DB subnet group and DB instance are in the same VPC.
Ok, it seems that I can't move to a different subnet in the same VPC, I started trying to create a new VPC, but it doesn't seem to be right and I'm sure there is something else I am missing here.
I also read about Network ACL, and thought that this might be the problem, but my rules seem to be fine, with the default rule to allow any traffic (and the rule * to DENY)
ALL Traffic ALL ALL 0.0.0.0/0 ALLOW
My RDS Network settings
Subnet group
default
Subnets
subnet-11111111
subnet-22222222
subnet-33333333
subnet-44444444
subnet-55555555
subnet-66666666
Security
VPC security groups
default (sg-111111)
( active )
Public accessibility
Yes
My Security group inbound rules
Type Protocol Port range Source Description - optional
All traffic All All 0.0.0.0/0 -
All traffic All All ::/0 -
Still can't connect, can't connect with my local client, can't even ping it:
Connecting through my local client
Can't connect to MySQL server on 'my-instance.xxxxxxxxxx.us-east-1.rds.amazonaws.com'
ping my-instance.xxxxxxx.us-east-1.rds.amazonaws.com
PING ec2-xx-xx-xx-xx.compute-1.amazonaws.com (xx.xx.xx.xx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Any idea of what I am missing here?
UPDATE
My VPC has internet access (I can access internet services from it, not an issue), I have an Internet Gateway and NAT Gateway in place.
I'm using Zappa for the lambda deployment, which takes care of creating a keep-warm function... however, I know that concurrent requests could still be an issue
The issue with VPC in lambda is that it can add 10s on the cold start, which is a no-deal for some of my use cases:
https://www.freecodecamp.org/news/lambda-vpc-cold-starts-a-latency-killer-5408323278dd/
Besides enabling "public access" on your RDS instance, you need to enable internet to the VPC using an internet gateway. After you attach it to the VPC, you need to route the data from the database subnets to the internet gateway. Check here
But I would not advise you expose your database like this. If you are having issues with lambda cold start, you should create an event to keep it warm.
Things you need to is :
Create new subnet group with default VPC
assign two subnet for availability zone
then modify your RDS instance
change subnet group to newly created group
mark "Publicly accessibility" to Yes.
Check your VPC is using internet gateway or not.
Check lambda security group whether it's open for outbound connection for Database port is not available or not.
No need to create different VPC for RDS. Use Default VPC.
As recommended by #stargazer try to not to expose publicly or out of VPC. Its works well inside VPC.
Related
I am trying to connect to my AWS RDS Mariadb instance that I am hosting on us-east/ohio from my local machine. I am trying to avoid making the instance publicly available but I am struggling to get this connection to work. Right now I am trying to connect from my local machine but eventually, I hope to host a nodejs server to talk to it on a static ip.
The setup I have now is the following:
A single VPC that my RDS is connected to which includes a CIDR that contains my public ip x.y.z.0/24
A route table which includes my public ip to connect to local
Network ACL inbound and outbound rule number 1 is to allow All TCP from 0.0.0.0/0
The Default security group which also allows all inbound and outbound traffic
A VPC endpoint attached to the RDS service
With all of this set up I figured it should allow anyone that has the DNS name of my VPC endpoint to talk to my RDS instance but I can not get a connection to my instance. I have used every DNS name associated with my endpoint and every single one of them times out when I try to sign into the database. I have been fumbling with this for days and would like to get past this point of initial setup.
Things possibly to note:
The Network ACL comes with a default rule of "*" deny all traffic. I do not know what order that rule is evaluated. I chose 1 for my rule of allowing all but I have also tried rule 100. Neither seems to work.
I know my RDS instance is on us-east-2a and I have made sure to add the us-east-2a subnet to my VPC endpoint. Using the DNS name that includes that at one point was giving me network unreachable for a little bit before I realized the subnet ID I chose was not the default which just gave me a timeout again.
I am trying to use DBeaver to connect to the VPC endpoint but I have also used the console command mysql -h vpce-<random characters>-<VPC ID>-us-east-2a.rsa.us-east-2.vpce.amazonaws.com -u admin -p and gotten the same timeout
RDS Database details:
I have an publicly accessible RDS database which is restricted to be accessible to only specific IP address.
The RDS have 2 security groups attached to the RDS:
1st security group allows a specific IP address to access it.
2nd security group allows multiple specific IP addresses to access
it.
Note: I know I can just keep a single security group for this purpose.
This RDS database have the default VPC and 3 public subnets. All these subnets have 2 routes in route tables:
Destination: default vpc's cidr range and Target: local
Destination: 0.0.0.0/0 and Target: internet gateway
Lambda function details:
I have a lambda function, which does not have any vpc configured.
I am trying to access the RDS database from code in the lambda function. Since lambda doesn't have any specific IP address, I couldn't add inbound rule in RDS's security group to allow it. When I simply add inbound rule in RDS's security group to allow access from all IPs then my lambda function code is able to connect to RDS database and work with it.
I wouldn't want my RDS database to be allowed access from anywhere on the internet, what configuration/settings do I need to make to be able to access RDS from lambda.
I also tried configuring the lambda function to be in same vpc and same subnets as that of for RDS instance; but I haven't been successful in making a connection to the RDS database.
Best in this case is to use the same VPC for your lambda as the RDS instance. This is cheaper and is more secure. You don't have to setup a internet gateway and elastic IP. And you can only access your RDS instance from within the VPC.
You can enable this using Lambda management console. Select Lambda function which need access to RDS instance and then go to Configuration -> Advanced settings and select the VPC (which is your RDS instance is in) you need it to access.
For instances attached to the same security group—make the security group the source for the inbound rule and the destination for the outbound rule.
For instances in different security groups—make sure that both security groups allow access to one another.
If this is still not working check the error message that is returned when setting up the connection with RDS. Timeout means issue with network.
find out more here https://aws.amazon.com/premiumsupport/knowledge-center/connect-lambda-to-an-rds-instance/
This question has been asked several times, and I've been reading the answer for the past hours, so I'm going to summarise everything here.
To get the connection between RDS and lambda, the normal steps are:
Attach the lambda to the same VPC as the database
Create a security group for the lambda
Modify / add a security group for the RDS, in which you allow the VPC's security group as inbound providing the connection port.
Add policy AWSLambdaVPCAccessExecutionRole to lambda function
That's supposed to be it, however it is not working for me.
I have checked other possible errors:
DNS of database not found. Solved by setting an internet gateway for the VPC, checked that the DNS is properly resolved
Inbound and Outbound rules of the lambda functions. Solved by setting literally all traffic for both directions.
Outbound rule for DB's security group is set to all traffic and destinations
Note: I can connect to the database via a VPN using another security group, meaning it's not a credential error.
I really don't understand what else I'm missing at this point
Finally found the answer, the error was due to the fact that the RDS was on a public subnet which had an internet gateway. In this case, the solution is as follows:
Create a public NAT gateway (attached to a public subnet, with an elastic ip)
Create one (or more) private subnets in the same VPC
Create a route with the “Destination” as “0.0.0.0”, “Target” as the Internet Gateway, and save your settings. Associate this route with the public subnets
Create a route in this table with the “Destination” as “0.0.0.0”, “Target” as the NAT Gateway created previously, and save your settings. Associate this route with the private subnets
Finally, connect the lambda to the private subnet(s) only
There are couple of questions around this, but none seem to solve my specific problem.
So i have a publicly accessible RDS instance in a VPC with two Security Groups : one the default SG for VPC & other to allow incoming connections from our developer machines. I am able to successfully connect to this RDS via my developer machines(personal laptops/PCs) as a result of the 2nd Security group.
Now, when i try to connect to this RDS via a Lambda running in same VPC, it fails with this error : "java.net.UnknownHostException: ****.rds.amazonaws.com: Name or service not known "
My lambda is getting launched in the same VPC as RDS and i have chosen all the RDS VPC subnets for the lambda. It also is tied to the default VPC security group.
Below is my understanding(obviously not right as connection is not working), what am i missing here?
The Default VPC SG(security group) associated to the lambda has outgoing rules set as 0.0.0.0/0 , hence it should allow lambda to access the whole world; so at least the RDS in VPC should be accessible.
The same default VPC SG should allow the RDS to accept incoming connections from any ip in the SG subnets(inbound rule mentions SG/default) ; hence lambda should be able to access it.
The error message is indicating that it cannot resolve the DNS Name of the RDS database (Name or service not known).
I suspect that this is due to the fact that the RDS instance has been configured as publicly accessible, which gives it a public DNS Name and IP address.
An AWS Lambda function connected to a VPC will send all network traffic via the VPC. It does not have access to the Internet unless there is a NAT in the VPC and the Lambda function is connected to a Private Subnet. Thus, the connection attempts from the Lambda function are possibly attempting to resolve a Public DNS Name, but is failing to do so.
This could likely be resolved by changing the RDS instance to be Publicly Available = No. This not only makes it safer, but it means that the DNS Name will resolve to a Private IP address, which should work correctly with a Lambda function connected to the VPC.
Alternatively, you could configure the Lambda function to not be connected to the VPC. This gives it Internet access, so it will connect to the RDS database just like any other computer on the Internet.
So the problem was that in my developer machine, I was using the db instance endpoint , but in lambda i was using the cluster endpoint. They look similar, hence i was not able to catch it earlier. On fixing the lambda to use the instance endpoint, both lambda & developer machines can successfully connect to the RDS.
However, I am now curious why the cluster url is not accessible, but i will create another question for that.
I have a target group which has two EC2 instances behind an application load balancer. I'm trying to build a lambda service to monitor EC2 instance healthy status outside of load balancer listener.
this lambda works as expected if VPC is not enabled. once I enabled VPC, then following code will fail:
client = boto3.client("elbv2", region_name=default_region_name)
tg_res = client.describe_target_groups(Names=[self.name])
API describe_target_groups will stuck there until timeout.
this VPC does have internet access and security group inbound/outbound temporary open for all IP.
any hit will be highly appreciated.
==========update==================
From lambda log, timeout happens at:
Starting new HTTPS connection (1): elasticloadbalancing.us-west-2.amazonaws.com
I checked network setting and I'm sure this VPC has NAT configuration.
this issue is solved by assigned another two internal access only subnets.
1) two EC2 instances are behind an ALB. EC2 instances are in subnet, let's say: west-2a-public, west-2b-public. These two subnets are internet facing and can assign public IP.
2) I assigned lambda service into same subnet: west-2a-public and west-2b-public but it doesn't work.
3) I then assigned lambda service into another two subnets: west-2a-private and west-2b-private. These two subnet does have NAT but doesn't have public IP assignment.
I guess the issue the router mapping but I need time to figure out exactly root cause. so far, it works.
You might need to edit your aws lambda function to add VPC support.
You can read more about here
I think the reason for this is the following: First, you need a NAT Gateway to access the Internet for Lambda functions as described here:
https://aws.amazon.com/de/blogs/aws/new-access-resources-in-a-vpc-from-your-lambda-functions/
Internet Access – As soon as you enable this functionality for a
particular function, the function no longer has access to the Internet
by default. If your function requires this type of access, you will
need to set up a Managed NAT Gateway in your VPC (see New – Managed
NAT (Network Address Translation) Gateway for AWS for more
information) or run your own NAT (see NAT Instances).
Second, if you have an EC2 instance or other Service such as a Load Balancer that need to be accessible inbound from the internet then you also need an Internet Gateway. However, it is not possible to assign an Internet Gateway and the NAT to one subnet. Therefore, you need to setup two subnets.
Here is an article describing the correct setup and configurations:
https://marcelog.github.io/articles/aws_lambda_internet_vpc.html