I'm trying to figure out the following:
I have an API service deployed in the default VPC, and I have a ELB configured to access the service.
The ELB is attached to a security group sg-XXXXXXXX1 to restrict inbound traffic (open to all for outbound)
I'm now trying to create a lambda function that can call the API service. (using python requests)
I've tried the following and failed to succeed:
In the configuration > VPC section, I added sg-XXXXXXXX1 as the security group and the 4 default subnets
Added AWSLambdaVPCAccessExecutionRole to lambda role
Created a new security group sg-XXXXXXXX2 (all traffic for inbound and outbound), attached it to lambda, and added sg-XXXXXXXX2 to the inbound list (all traffic) for sg-XXXXXXXX1
Any ideas on what I did wrong? and how I can fix it?
Your ELB is most likely a public ELB (which is the default setting). The ELB DNS will resolve to a public IP address, not a VPC IP address. That means that the traffic going to the ELB will exit the VPC, go out to the Internet, and then back into AWS and into the ELB. When that happens, any association with the Lambda function's security group gets lost. Also, if your Lambda function isn't running in a subnet with a route to a NAT gateway it won't be able to access the Internet at all, so it is just going to timeout and fail when it tries to access the ELB.
To fix this, change the load balancer scheme to private, which will give the load balancer a private VPC IP address, and make it accessible only within the VPC.
Related
I have a full AWS HTTPS web service, with all needed components i.e. a VPC containing:
private ec2 instances
autoscaling groups
a load balancer (with a public domain xxxxx.eu-west-1.elb.amazonaws.com , and even an official public domain xxxxx.com )
security groups
All of this works, and I can access from outside the Amazon cloud to xxxxx.com (using my Golang HTTP client code based on "net/http", for example), provided that I put my client IP address in the inbound rules of the security group of my load balancer (I filter IPs because it's a B2B service so I don't want to let anybody come in).
I have also a lambda function, and I would like to access this web service from that lambda function. The problem is, lambda functions don't have stable IP addresses. My request is similar to this one except that I don't want to access an EC2 instance directly (that would be unadvised since the service is scalable), simply access the public service like somebody from outside the Amazon cloud.
Currently, my lambda function can access any website on the Internet, except my HTTP web service. When I access my HTTP web service my Golang lambda client (the same code than above, so it's not a client issue) hangs in this function call, until the lambda timeouts (or my client timeouts if I configure a timeout in the client):
response, err = client.Do(req) // <--- hangs (client is a http.Client)
I tried to apply the recommended solution:
Allow all (0.0.0.0/0) in the inbound rules of the VPC security group. It works, but like I said, I want to eventually filter IPs to allow only specific clients (in addition of my lambda).
Add the lambda function in the VPC (the same VPC that the web service), create a security group for the lambda (with no inbounds rules, all/default outbounds rules), and allow that security group into in the inbound rules of the VPC security group. It doesn't work for some reason.
Is there a solution to do what I want?
Add the lambda function in the VPC (the same VPC that the web
service), create a security group for the lambda (with no inbounds
rules, all/default outbounds rules), and allow that security group
into in the inbound rules of the VPC security group. It doesn't work
for some reason.
You are on the right track here, but unfortunately your public load balancer will be resolved by the Lambda function to its public IP address, which exists outside the VPC. So the Lambda then tries to reach out of the VPC to access the load balancer, at which point the traffic becomes disassociated with the Lambda function's security group.
You have two options:
Add an internal load balancer to your infrastructure, and a Route53 private hosted zone to resolve the domain name to the internal load balancer from within your VPC.
Deploy the Lambda function in VPC subnets that have a route to a NAT Gateway, and allow the NAT Gateway's Elastic IP address in the public load balancer's security group.
I have been struggling with this problem for 2 days but couldn't get it working.
I have this flow:
external world --> AWS API Gateway ---> VPC Link ---> Network Load Balancer ---> my single EC2 instance
Before introducing the API Gateway, I want to first make sure the Network Load Balancer --> my single EC2 instance part works.
I have set up the EC2 instance correctly. There is a Typescript / ExpressJS api service running on port 3001
I have also set up a Network Load Balancer and a Target Group, the NLB is listening and forwarding requests to port 3001 of the target group (which contains the EC2 instance).
Here is the NLB:
Note that the NLB has a VPC! This raise the question below and I find it so confusing.
listener:
You can see it is forwarding requests to docloud-backend-service, which is described as follows:
You can see that the health check has passed.
I have configured the security group of my EC2 instance with this rule:
1. Allow All protocol traffic on All ports from my VPC
(specified using CIDR notation `171.23.0.0/16`);
Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user, the command fails by timeout.
Then, after I add this rule:
2. Allow All protocol traffic on All ports from ANY source (`0.0.0.0/0`);
Now, when I do curl docloud-backend-xxxxx.elb.ap-northeast-1.amazonaws.com:3001/api/user,
the api service gets the request and I can see logs generated in the EC2 instance.
Question:
The second rule opens up the EC2 instance to public, which is dangerous.
I want to limit access to my EC2 instance port 3001 such that only the AWS API Gateway, or the NLB can access it.
The NLB has no security group to be configured. It has a VPC though. If I limit the EC2 instance such that only its own VPC can access it, it should be fine, right?
The first rule does exactly that. Why does it fail?
The NLB has a VPC. Requests go from API Gateway to NLB, then from NLB to EC2 instance. So from the EC2 instance's perspective, the requests come from an entity in the VPC. So the first rule should work, right?
Otherwise why would AWS assign a VPC to the NLB anyways?
Why would I see the VPC on the NLB's description console anyways?
I want to limit access to my EC2 instance port 3001 such that only the AWS API Gateway, or the NLB can access it.
For instance based target groups and for IP based target groups as well we can enable/disable if want to preserve the requester's IP address:
This setting can be found if go to our target group -> Actions -> Edit Target attributes.
What does this mean from the perspective of the Security Group of our application?
If we enable it (which is the default for instance type target groups), the application will see traffic as it is coming directly from the end-client. This means, you we have to enable inbound traffic for 0.0.0.0:3001.
If we disable it, the application will see the source traffic as it was coming from the private IP address of the Network Load Balancer. In this case, we can limit the inbound traffic to the private IP address of the NLB or to the CIDR range of the subnet in which the NLB is placed.
This question has been asked several times, and I've been reading the answer for the past hours, so I'm going to summarise everything here.
To get the connection between RDS and lambda, the normal steps are:
Attach the lambda to the same VPC as the database
Create a security group for the lambda
Modify / add a security group for the RDS, in which you allow the VPC's security group as inbound providing the connection port.
Add policy AWSLambdaVPCAccessExecutionRole to lambda function
That's supposed to be it, however it is not working for me.
I have checked other possible errors:
DNS of database not found. Solved by setting an internet gateway for the VPC, checked that the DNS is properly resolved
Inbound and Outbound rules of the lambda functions. Solved by setting literally all traffic for both directions.
Outbound rule for DB's security group is set to all traffic and destinations
Note: I can connect to the database via a VPN using another security group, meaning it's not a credential error.
I really don't understand what else I'm missing at this point
Finally found the answer, the error was due to the fact that the RDS was on a public subnet which had an internet gateway. In this case, the solution is as follows:
Create a public NAT gateway (attached to a public subnet, with an elastic ip)
Create one (or more) private subnets in the same VPC
Create a route with the “Destination” as “0.0.0.0”, “Target” as the Internet Gateway, and save your settings. Associate this route with the public subnets
Create a route in this table with the “Destination” as “0.0.0.0”, “Target” as the NAT Gateway created previously, and save your settings. Associate this route with the private subnets
Finally, connect the lambda to the private subnet(s) only
I would like to create a lambda(vpc) which would access resources in vpc and make a request to services(REST API) via public application load balancer. I found out that vpc end point is better solution than creating a nat gateway.
I have created a vpc endpoint for elasticloadbalancing(by following steps at https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) and given full access in the policy. I could not find how to access it from the lambda, what would be the URL to make the request?
Edit:
Thanks to John for the info that vpc endpoint is used to connect to ELB API. So Vpc endpoint would not solve our issue.
We have our infra in vpc which includes database(accessible within vpc only) and application servers running behind the ELB. For certain tasks we want to run lambda which will read database(for this reason lambda has to be inside vpc) and make API calls to our application using ELB. Since ELB is accessible from public dns only, lambda is not able to connect to ELB.
I have read that setting up NAT gateway is a solution. Are there other is simpler ways?
Yes, a NAT Gateway would allow the traffic from a private subnet to go out of the VPC and come back in to the Load Balancer's public IP addresses (via its Public DNS Name).
Alternatively, you could create an additional Internal Load Balancer that could accept traffic from within the VPC and send it to the Amazon EC2 instances.
I have a target group which has two EC2 instances behind an application load balancer. I'm trying to build a lambda service to monitor EC2 instance healthy status outside of load balancer listener.
this lambda works as expected if VPC is not enabled. once I enabled VPC, then following code will fail:
client = boto3.client("elbv2", region_name=default_region_name)
tg_res = client.describe_target_groups(Names=[self.name])
API describe_target_groups will stuck there until timeout.
this VPC does have internet access and security group inbound/outbound temporary open for all IP.
any hit will be highly appreciated.
==========update==================
From lambda log, timeout happens at:
Starting new HTTPS connection (1): elasticloadbalancing.us-west-2.amazonaws.com
I checked network setting and I'm sure this VPC has NAT configuration.
this issue is solved by assigned another two internal access only subnets.
1) two EC2 instances are behind an ALB. EC2 instances are in subnet, let's say: west-2a-public, west-2b-public. These two subnets are internet facing and can assign public IP.
2) I assigned lambda service into same subnet: west-2a-public and west-2b-public but it doesn't work.
3) I then assigned lambda service into another two subnets: west-2a-private and west-2b-private. These two subnet does have NAT but doesn't have public IP assignment.
I guess the issue the router mapping but I need time to figure out exactly root cause. so far, it works.
You might need to edit your aws lambda function to add VPC support.
You can read more about here
I think the reason for this is the following: First, you need a NAT Gateway to access the Internet for Lambda functions as described here:
https://aws.amazon.com/de/blogs/aws/new-access-resources-in-a-vpc-from-your-lambda-functions/
Internet Access – As soon as you enable this functionality for a
particular function, the function no longer has access to the Internet
by default. If your function requires this type of access, you will
need to set up a Managed NAT Gateway in your VPC (see New – Managed
NAT (Network Address Translation) Gateway for AWS for more
information) or run your own NAT (see NAT Instances).
Second, if you have an EC2 instance or other Service such as a Load Balancer that need to be accessible inbound from the internet then you also need an Internet Gateway. However, it is not possible to assign an Internet Gateway and the NAT to one subnet. Therefore, you need to setup two subnets.
Here is an article describing the correct setup and configurations:
https://marcelog.github.io/articles/aws_lambda_internet_vpc.html