I have a full AWS HTTPS web service, with all needed components i.e. a VPC containing:
private ec2 instances
autoscaling groups
a load balancer (with a public domain xxxxx.eu-west-1.elb.amazonaws.com , and even an official public domain xxxxx.com )
security groups
All of this works, and I can access from outside the Amazon cloud to xxxxx.com (using my Golang HTTP client code based on "net/http", for example), provided that I put my client IP address in the inbound rules of the security group of my load balancer (I filter IPs because it's a B2B service so I don't want to let anybody come in).
I have also a lambda function, and I would like to access this web service from that lambda function. The problem is, lambda functions don't have stable IP addresses. My request is similar to this one except that I don't want to access an EC2 instance directly (that would be unadvised since the service is scalable), simply access the public service like somebody from outside the Amazon cloud.
Currently, my lambda function can access any website on the Internet, except my HTTP web service. When I access my HTTP web service my Golang lambda client (the same code than above, so it's not a client issue) hangs in this function call, until the lambda timeouts (or my client timeouts if I configure a timeout in the client):
response, err = client.Do(req) // <--- hangs (client is a http.Client)
I tried to apply the recommended solution:
Allow all (0.0.0.0/0) in the inbound rules of the VPC security group. It works, but like I said, I want to eventually filter IPs to allow only specific clients (in addition of my lambda).
Add the lambda function in the VPC (the same VPC that the web service), create a security group for the lambda (with no inbounds rules, all/default outbounds rules), and allow that security group into in the inbound rules of the VPC security group. It doesn't work for some reason.
Is there a solution to do what I want?
Add the lambda function in the VPC (the same VPC that the web
service), create a security group for the lambda (with no inbounds
rules, all/default outbounds rules), and allow that security group
into in the inbound rules of the VPC security group. It doesn't work
for some reason.
You are on the right track here, but unfortunately your public load balancer will be resolved by the Lambda function to its public IP address, which exists outside the VPC. So the Lambda then tries to reach out of the VPC to access the load balancer, at which point the traffic becomes disassociated with the Lambda function's security group.
You have two options:
Add an internal load balancer to your infrastructure, and a Route53 private hosted zone to resolve the domain name to the internal load balancer from within your VPC.
Deploy the Lambda function in VPC subnets that have a route to a NAT Gateway, and allow the NAT Gateway's Elastic IP address in the public load balancer's security group.
Related
I'm trying to figure out the following:
I have an API service deployed in the default VPC, and I have a ELB configured to access the service.
The ELB is attached to a security group sg-XXXXXXXX1 to restrict inbound traffic (open to all for outbound)
I'm now trying to create a lambda function that can call the API service. (using python requests)
I've tried the following and failed to succeed:
In the configuration > VPC section, I added sg-XXXXXXXX1 as the security group and the 4 default subnets
Added AWSLambdaVPCAccessExecutionRole to lambda role
Created a new security group sg-XXXXXXXX2 (all traffic for inbound and outbound), attached it to lambda, and added sg-XXXXXXXX2 to the inbound list (all traffic) for sg-XXXXXXXX1
Any ideas on what I did wrong? and how I can fix it?
Your ELB is most likely a public ELB (which is the default setting). The ELB DNS will resolve to a public IP address, not a VPC IP address. That means that the traffic going to the ELB will exit the VPC, go out to the Internet, and then back into AWS and into the ELB. When that happens, any association with the Lambda function's security group gets lost. Also, if your Lambda function isn't running in a subnet with a route to a NAT gateway it won't be able to access the Internet at all, so it is just going to timeout and fail when it tries to access the ELB.
To fix this, change the load balancer scheme to private, which will give the load balancer a private VPC IP address, and make it accessible only within the VPC.
When I configure an AWS Gateway VPC endpoint, a route table entry is created that points to the Gateway. Here, Gateway can be thought of performing the routing to AWS service (over private network).
However, for an AWS Inteface VPC endpoint, all that is visible is a Network interface that has a private IP address of the subnet. By default, a private IP can send traffic within the subnet or entire VPC provided Security Group and NACL allows the traffic. & it appears in this case there is no Route table entry to a Gateway or a Router for allowing traffic outside VPC.
How / Where is the interface routing the traffic to i.e. How does traffic leave the customer VPC?
Of course I understand that the traffic finally reaches the intended AWS service over private network but here I am trying to find out where is the Gateway or Router? Does AWS hide this implementation?
I cannot get my head around the fact that a simple Network Interface can accept traffic and route it to a service all by itself i.e. performing routing by itself? Clearly, in this case the traffic appears not flowing through the VPC router or another Gateway device.
I am aware this might be an AWS confidential implementation but any thoughts / idea on how they might have designed this feature?
It doesn't provide routing at all, by default a VPC interface endpoint when created will create an ENI per subnet in the VPC for you. It will also provide you a DNS name per each AZ and a global name that you can use within your applications.
In addition it supports the ability to have the AWS service domain name for the VPC interface endpoint be resolvable to the private IPs of the endpoint. As long as your VPC has DNS enabled it will first check the VPC private DNS resolver and then resolve it to the private IP rather than the public one.
This is done by adding an additional private hosted zone to your VPC which resolves service domains in your region such as ec2.us-east-1.amazonaws.com.
From the AWS side this is just an ENI created in your AWS VPC that is connected to one of AWS internal VPCs. It's actually possible to implement this for your own services too to share with another organisations VPCs, this is implemented using AWS PrivateLink.
For more information take a look at the Private DNS for interface endpoints page.
I have a simply maked it with serverless endpoint for check some token. Lets say the path is "/checktoken". And want to access to it only with my Ip address, thats why connect my Lambda with VPC(2 private subnets and security group).
But problem is even I maked security group in/outbound rules as https with my IP address, I can request to the endpoint with different IP address.
I have used AWS = CloudFront + Route53 for DNS, Lambda + Api Gateway for endpoint.
I made a whitelist IP with resource policy in Api gateway which works, but its not the solution I want.
The required solution is vpc's security group should allow request only from whitelisted IP in its rule.
Adding the Lambda to a VPC will not direct traffic to the Lambda through your VPC.
In fact The Lambda is invoked through the Lambda API Service Endpoint, therefore inbound evaluation rules will have no effect on it at all. The purpose of a Lambda residing in a VPC is to access VPC resources.
The technical implementation is that an ENI is created in your VPC, that connects to the Lambda function in a shared AWS VPC. The Lambda is able to route out of its shared VPC to connect to resources.
When you configure your Lambda function to connect to your own VPC, it creates an elastic network interface in your VPC and then does a cross-account attachment. These network interfaces allow network access from your Lambda functions to your private resources. These Lambda functions continue to run inside of the Lambda service’s VPC and can now only access resources over the network through your VPC.
You will need to add these IP whitelists at either the CloudFront level through the use of a AWS WAF (using an IPSet), or through the API Gateway as a WAF or Policy (as you mentioned above).
thats why connect my Lambda with VPC(2private subnets and security group)
Placing lambda function in a VPC and giving it a security group does not have effect on whether the API gateway can invoke it or not.
API gateway does not use elastic network interface (ENI) of the lambda in the VPC to invoke it. Its done using Lambda service endpoint.
You can use WAF to control access to your CloudFront distribution:
Using AWS WAF to Control Access to Your Content
Try Black / White listing IPAdresses:
https://www.npmjs.com/package/serverless-secure
I would like to create a lambda(vpc) which would access resources in vpc and make a request to services(REST API) via public application load balancer. I found out that vpc end point is better solution than creating a nat gateway.
I have created a vpc endpoint for elasticloadbalancing(by following steps at https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint) and given full access in the policy. I could not find how to access it from the lambda, what would be the URL to make the request?
Edit:
Thanks to John for the info that vpc endpoint is used to connect to ELB API. So Vpc endpoint would not solve our issue.
We have our infra in vpc which includes database(accessible within vpc only) and application servers running behind the ELB. For certain tasks we want to run lambda which will read database(for this reason lambda has to be inside vpc) and make API calls to our application using ELB. Since ELB is accessible from public dns only, lambda is not able to connect to ELB.
I have read that setting up NAT gateway is a solution. Are there other is simpler ways?
Yes, a NAT Gateway would allow the traffic from a private subnet to go out of the VPC and come back in to the Load Balancer's public IP addresses (via its Public DNS Name).
Alternatively, you could create an additional Internal Load Balancer that could accept traffic from within the VPC and send it to the Amazon EC2 instances.
I have a target group which has two EC2 instances behind an application load balancer. I'm trying to build a lambda service to monitor EC2 instance healthy status outside of load balancer listener.
this lambda works as expected if VPC is not enabled. once I enabled VPC, then following code will fail:
client = boto3.client("elbv2", region_name=default_region_name)
tg_res = client.describe_target_groups(Names=[self.name])
API describe_target_groups will stuck there until timeout.
this VPC does have internet access and security group inbound/outbound temporary open for all IP.
any hit will be highly appreciated.
==========update==================
From lambda log, timeout happens at:
Starting new HTTPS connection (1): elasticloadbalancing.us-west-2.amazonaws.com
I checked network setting and I'm sure this VPC has NAT configuration.
this issue is solved by assigned another two internal access only subnets.
1) two EC2 instances are behind an ALB. EC2 instances are in subnet, let's say: west-2a-public, west-2b-public. These two subnets are internet facing and can assign public IP.
2) I assigned lambda service into same subnet: west-2a-public and west-2b-public but it doesn't work.
3) I then assigned lambda service into another two subnets: west-2a-private and west-2b-private. These two subnet does have NAT but doesn't have public IP assignment.
I guess the issue the router mapping but I need time to figure out exactly root cause. so far, it works.
You might need to edit your aws lambda function to add VPC support.
You can read more about here
I think the reason for this is the following: First, you need a NAT Gateway to access the Internet for Lambda functions as described here:
https://aws.amazon.com/de/blogs/aws/new-access-resources-in-a-vpc-from-your-lambda-functions/
Internet Access – As soon as you enable this functionality for a
particular function, the function no longer has access to the Internet
by default. If your function requires this type of access, you will
need to set up a Managed NAT Gateway in your VPC (see New – Managed
NAT (Network Address Translation) Gateway for AWS for more
information) or run your own NAT (see NAT Instances).
Second, if you have an EC2 instance or other Service such as a Load Balancer that need to be accessible inbound from the internet then you also need an Internet Gateway. However, it is not possible to assign an Internet Gateway and the NAT to one subnet. Therefore, you need to setup two subnets.
Here is an article describing the correct setup and configurations:
https://marcelog.github.io/articles/aws_lambda_internet_vpc.html