I have a containerized Lambda deployed that uses the latest image from ECR. I have also already setup the networking with the following:
Subnets
Route Table
private-1.
private-route
private-2.
private-route
public-0.
public-route.
Subnet CIDRs
private-1: 172.31.32.0/20
private-2: 172.31.48.0/20
Private Route Table
Destination
Target
172.31.0.0/16
local.
0.0.0.0/0.
nat-xxx
Public Route Table
Destination
Target
172.31.0.0/16
local.
0.0.0.0/0.
igw-xxx
The Nat Gateway is associatd with the public-1 subnet.
Security Group
All Traffic Inbound and Outbound is allowed for now (for debugging).
Lambda Configuration
Subnets.
Security Groups
private 1 & 2
sg-xxx.
My problem is after I set up this configuration I was able to get access, then I added configuration to talk to a peered vpc for a database connection and that also worked. But immediately following It stopped working. So my confusion is why is it sporadic? I'll randomly change security groups and redo networking and it works once then stops. This is the error I constantly see now which is the first few lines of the lambda invocation.
"errorMessage": "HTTPSConnectionPool(host='maps.googleapis.com', port=443): Max retries exceeded with url: /maps/api/geocode/json?
Related
I'm trying to figure out the following:
I have an API service deployed in the default VPC, and I have a ELB configured to access the service.
The ELB is attached to a security group sg-XXXXXXXX1 to restrict inbound traffic (open to all for outbound)
I'm now trying to create a lambda function that can call the API service. (using python requests)
I've tried the following and failed to succeed:
In the configuration > VPC section, I added sg-XXXXXXXX1 as the security group and the 4 default subnets
Added AWSLambdaVPCAccessExecutionRole to lambda role
Created a new security group sg-XXXXXXXX2 (all traffic for inbound and outbound), attached it to lambda, and added sg-XXXXXXXX2 to the inbound list (all traffic) for sg-XXXXXXXX1
Any ideas on what I did wrong? and how I can fix it?
Your ELB is most likely a public ELB (which is the default setting). The ELB DNS will resolve to a public IP address, not a VPC IP address. That means that the traffic going to the ELB will exit the VPC, go out to the Internet, and then back into AWS and into the ELB. When that happens, any association with the Lambda function's security group gets lost. Also, if your Lambda function isn't running in a subnet with a route to a NAT gateway it won't be able to access the Internet at all, so it is just going to timeout and fail when it tries to access the ELB.
To fix this, change the load balancer scheme to private, which will give the load balancer a private VPC IP address, and make it accessible only within the VPC.
I have been struggling with this problem for 2 days but couldn't get it working.
I have this flow:
external world --> AWS API Gateway ---> VPC Link ---> Network Load Balancer ---> my single EC2 instance
Before introducing the API Gateway, I want to first make sure the Network Load Balancer --> my single EC2 instance part works.
I have set up the EC2 instance correctly. There is a Typescript / ExpressJS api service running on port 3001
I have also set up a Network Load Balancer and a Target Group, the NLB is listening and forwarding requests to port 3001 of the target group (which contains the EC2 instance).
Here is the NLB:
Note that the NLB has a VPC! This raise the question below and I find it so confusing.
listener:
You can see it is forwarding requests to docloud-backend-service, which is described as follows:
You can see that the health check has passed.
I have configured the security group of my EC2 instance with this rule:
1. Allow All protocol traffic on All ports from my VPC
(specified using CIDR notation `171.23.0.0/16`);
Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user, the command fails by timeout.
Then, after I add this rule:
2. Allow All protocol traffic on All ports from ANY source (`0.0.0.0/0`);
Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user,
the api service gets the request and I can see logs generated in the EC2 instance.
Question:
The second rule opens up the EC2 instance to public, which is dangerous.
I want to limit access to my EC2 instance port 3001 such that only the AWS API Gateway, or the NLB can access it.
The NLB has no security group to be configured. It has a VPC though. If I limit the EC2 instance such that only its own VPC can access it, it should be fine, right?
The first rule does exactly that. Why does it fail?
The NLB has a VPC. Requests go from API Gateway to NLB, then from NLB to EC2 instance. So from the EC2 instance's perspective, the requests come from an entity in the VPC. So the first rule should work, right?
Otherwise why would AWS assign a VPC to the NLB anyways?
Why would I see the VPC on the NLB's description console anyways?
I want to limit access to my EC2 instance port 3001 such that only the AWS API Gateway, or the NLB can access it.
For instance based target groups and for IP based target groups as well we can enable/disable if want to preserve the requester's IP address:
This setting can be found if go to our target group -> Actions -> Edit Target attributes.
What does this mean from the perspective of the Security Group of our application?
If we enable it (which is the default for instance type target groups), the application will see traffic as it is coming directly from the end-client. This means, you we have to enable inbound traffic for 0.0.0.0:3001.
If we disable it, the application will see the source traffic as it was coming from the private IP address of the Network Load Balancer. In this case, we can limit the inbound traffic to the private IP address of the NLB or to the CIDR range of the subnet in which the NLB is placed.
This question has been asked several times, and I've been reading the answer for the past hours, so I'm going to summarise everything here.
To get the connection between RDS and lambda, the normal steps are:
Attach the lambda to the same VPC as the database
Create a security group for the lambda
Modify / add a security group for the RDS, in which you allow the VPC's security group as inbound providing the connection port.
Add policy AWSLambdaVPCAccessExecutionRole to lambda function
That's supposed to be it, however it is not working for me.
I have checked other possible errors:
DNS of database not found. Solved by setting an internet gateway for the VPC, checked that the DNS is properly resolved
Inbound and Outbound rules of the lambda functions. Solved by setting literally all traffic for both directions.
Outbound rule for DB's security group is set to all traffic and destinations
Note: I can connect to the database via a VPN using another security group, meaning it's not a credential error.
I really don't understand what else I'm missing at this point
Finally found the answer, the error was due to the fact that the RDS was on a public subnet which had an internet gateway. In this case, the solution is as follows:
Create a public NAT gateway (attached to a public subnet, with an elastic ip)
Create one (or more) private subnets in the same VPC
Create a route with the “Destination” as “0.0.0.0”, “Target” as the Internet Gateway, and save your settings. Associate this route with the public subnets
Create a route in this table with the “Destination” as “0.0.0.0”, “Target” as the NAT Gateway created previously, and save your settings. Associate this route with the private subnets
Finally, connect the lambda to the private subnet(s) only
This is a follow up to my previous question. I'm successfully able to access the public IP over fargate. However, after trying to attach Application load balancer, I cannot access application over public DNS. The registered target always show unhealthy status with 502 error.
Sharing my configuration settings-
VPC
Route Table Public Subnet
Route Table public subnet config
Route Table private subnet
Route Table private subnet config
ACL subnet associations
ACL inbound rules
ACL outbound rules
security inbound rules
security outbound rules
internet gateway
nat gateway
target group
target group health status
task details
service
ALB settings
ALB listeners
Target group with 8081 port
Based on the comments.
There were two issues identified:
Incorrect port on the target group. It was 80 instead of 8081.
Healthy threshold timeout was too short (5 seconds). Increasing it seemed to solved the issue.
I was running a serverless web application on a lambda inside a VPC, and connecting to a Aurora-MySQL RDS instance, with inbound rules to allow traffic from the security group of the lambda
The connection was working fine, however, quite often the lambda cold start was giving me a timeout.
After some research, I found out that running a lambda on a VPC brings an additional cost on startup and I saw the recommendation in more than 1 place to avoid using lambda on a VPC except if you strictly need to access some resource in the VPC.
So, I decided to move my RDS to a publicly accessible instance, so my lambda can access it over the internet and remove the lambda from the VPC.
So, I changed the RDS Public accessibility option to Yes and edited the security group to allow inbound connection from any IP.
I have also removed the VPC from the lambda, so the lambda is not running on a VPC anymore
I thought it was gonna be enough.
But then my lambda started failing to connect to the database
I tried to connect using my local client, again, failure
tried pinging to the hostname, got request timeouts
After digging a bit into it, I found that my DB instance subnet group having some private subnets might be a problem (?)
So, I have created a new subnet group with only public subnets, and tried to move my db instance to the new subnet group... but got this message:
You cannot move DB instance my-instance to subnet group my-new-group. The specified DB subnet group and DB instance are in the same VPC.
Ok, it seems that I can't move to a different subnet in the same VPC, I started trying to create a new VPC, but it doesn't seem to be right and I'm sure there is something else I am missing here.
I also read about Network ACL, and thought that this might be the problem, but my rules seem to be fine, with the default rule to allow any traffic (and the rule * to DENY)
ALL Traffic ALL ALL 0.0.0.0/0 ALLOW
My RDS Network settings
Subnet group
default
Subnets
subnet-11111111
subnet-22222222
subnet-33333333
subnet-44444444
subnet-55555555
subnet-66666666
Security
VPC security groups
default (sg-111111)
( active )
Public accessibility
Yes
My Security group inbound rules
Type Protocol Port range Source Description - optional
All traffic All All 0.0.0.0/0 -
All traffic All All ::/0 -
Still can't connect, can't connect with my local client, can't even ping it:
Connecting through my local client
Can't connect to MySQL server on 'my-instance.xxxxxxxxxx.us-east-1.rds.amazonaws.com'
ping my-instance.xxxxxxx.us-east-1.rds.amazonaws.com
PING ec2-xx-xx-xx-xx.compute-1.amazonaws.com (xx.xx.xx.xx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Any idea of what I am missing here?
UPDATE
My VPC has internet access (I can access internet services from it, not an issue), I have an Internet Gateway and NAT Gateway in place.
I'm using Zappa for the lambda deployment, which takes care of creating a keep-warm function... however, I know that concurrent requests could still be an issue
The issue with VPC in lambda is that it can add 10s on the cold start, which is a no-deal for some of my use cases:
https://www.freecodecamp.org/news/lambda-vpc-cold-starts-a-latency-killer-5408323278dd/
Besides enabling "public access" on your RDS instance, you need to enable internet to the VPC using an internet gateway. After you attach it to the VPC, you need to route the data from the database subnets to the internet gateway. Check here
But I would not advise you expose your database like this. If you are having issues with lambda cold start, you should create an event to keep it warm.
Things you need to is :
Create new subnet group with default VPC
assign two subnet for availability zone
then modify your RDS instance
change subnet group to newly created group
mark "Publicly accessibility" to Yes.
Check your VPC is using internet gateway or not.
Check lambda security group whether it's open for outbound connection for Database port is not available or not.
No need to create different VPC for RDS. Use Default VPC.
As recommended by #stargazer try to not to expose publicly or out of VPC. Its works well inside VPC.