I have a Lambda running within a VPC which accesses secure resources (ex: RDS), but I also need it to publish an SNS notification. Is there a way to do this without having a NAT gateway?
Alternatively, I'm thinking of writing to a DynamoDB table which triggers another lambda but wanted to know if there's a simpler approach.
The simple answer is no. SNS is not currently available as a VPC endpoint so you will need to continue doing what you are already doing in order to reach RDS via lambda (NAT gateway in the private subnet).
In other words, this answer from 2016 is still relevant today -> How to let AWS lambda in a VPC to publish SNS notification?
Option A: using a NAT gateway
Your lambda is in a private subnet which means it cannot have communication with the outside world (the internet), so unless you make that a public subnet, which is of course not recommended, you cannot access the outside world. A NAT gateway allows your resources to have that access through it, and it's really not that hard to implement.
Here's a handy tutorial on how to do that:
https://github.com/naguibihab/aws-goodies/blob/master/how-to-setup-lambda-to-talk-to-internet-and-vpc.md
Option B: using a NAT instance Similar to using a NAT gateway, you can also use a NAT instance which requires a bit more administration and might have less availability but it can also be cheaper. A NAT instance works in the same way as a NAT gateway, it needs to sit in a public subnet and any lambda functions in the private subnet can access the internet through it.
As for your DynamoDB alternative, you can create a VPC Endpoint for Dynamo db as Khalid T suggested: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints-ddb.html
Edit
I have been corrected, edited my answer.
Related
I am connecting to AWS DocumentDB from a Lambda function. In order to be able to do this I had to attach lambda to the default VPC (that's where DocumentDB cluster is running) and the default (public) subnets. But, this has caused my Lambda to timeout whenever trying make an outbound request, e.g. push message to SQS. This, I want to avoid.
So what is the recommended way of connecting to DocumentDB without loosing functionality that occurs when putting Lambda in the VPC? There's gotta be a simple solution.
Lambda functions in a VPC never get a public IP address. So if the function needs to access both VPC resources and other resources outside of the VPC the function has to be deployed only to private subnets with routes to a NAT Gateway.
Alternatively, if the only external resources you need to access are other AWS services, then you could add VPC Endpoints for those services to the VPC.
So I have a very confusing issue that I don't know how to solve. My setup is API Gateway -> Lambda -> IoT Core. I setup the code and it works fine from my IDE. I deploy it to AWS, and my connection to AWS times out.
The Lambda is in a single subnet and the subnet does have a default route to an IGW. I did a test, and the Lambda function can resolve the IP of my IoT endpoint to a public IP (54.x.x.x). But the connect() method times out. My security group for the Lambda function is setup to allow all incoming / outgoing.
What am I missing? Why can't I get to IoT Core from inside a VPC with an IGW configured and seems to be working. Any direction would be greatly appreciated.
UPDATE
After playing around with many different things, I can't identify what exactly I had messed up with my configuration. But following the accepted answer I ended up with the following setup which appears to work for what I need.
subnet-1 10.14.10.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-2 10.14.20.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-3 10.14.30.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-4 10.14.40.0/24 (auto-assign-public=false)
local route ( 10.14.0.0/16 ) and default route=nat-gateway
subnet-5 10.14.200.0/24 (auto-assign-public=true)
local route ( 10.14.0.0/16 ) and default route=igw
nat-gateway
in subnet-5
I don't know if this is what I intended, but this is what I was looking for. A series of subnets that are not publicly accessible, but has an internet connection for access to other AWS services. So my Lambda resources, ECS, etc can sit privately and access what they need.
Thank you everyone for the information.
You should not deploy the Lambda function to a public subnet (that's the subnet with the default route to the IGW). It won't work the way you want it to work. The Lambda function doesn't have, and cannot have, a public IP so cannot route to the internet via the IGW.
If the Lambda needs to be in VPC, then move it to a private subnet and make sure that the private subnet has a default route to a NAT (or NAT gateway) in a public subnet. Or deploy the Lambda function outside of VPC completely, if that's viable.
More information at:
Why can't Lambda in public subnet reach the internet?
Why do we need private subnet in a VPC?
When you say "I did a test, and the Lambda function can resolve the IP of my IoT endpoint to a public IP (54.x.x.x)" Do you mean DNS resolution, or you've checked this with a actual network traffic.
In either case, you can turn VPC Flow Logs for your VPC, and try again. The flow log will identify whether SGs or NACLs are blocking your traffic.
Remember also that Lambda's cannot exists in a public subnet, they have to reside in private subnets, and use NAT GW on public subnets to connect to the internet.
I encountered the same issue. Thankfully, AWS has automated this process and it's only a few clicks but sadly an extra $$$/month (about $30 to start).
You need to create a VPC group that has both public and private subnets. For my case, I was accessing a RDS database to then build email templates and fire off these emails via SES.
What did not work:
When I deployed a default VPC on my lambda function, the RDS would work, but the SES would not. The reason being is the SES API has no access to the internet via default VPCs, you need to setup a private subnet to allow for this.
VPC wizard can do this all for you with a few simple clicks, but you're looking at spending about $30/month (CAD) to setup a NAT gateway + any potential data processing charges per GB (read the pricing). For my case, firing a few email templates off it outweighed deploying another ec2 instance and gave me flexibility for scaling without too much on-the-fly configuration.
This link is a good place to get started. How do I give internet access to a Lambda function that's connected to an Amazon VPC?
Follow the VPC wizard hyperlink within that article, and you just need to go to the VPC console and create a VPC, should look something like this (below)
Once AWS works its magic, you need to assign the VPC to your Lambda function, assign the right permissions to your lambda function AND don't forget to configure your security group with the correct inbound rules. In my case I need to add a PostGres rule to access my RDS.
Hope this helps!
I have a lambda function which runs every 15 minutes and saves some data in DynamoDB.
Now I want to secure the DynamoDB call made by my lambda so that the request does not go via the Internet, rather through Amazon internal network. There is no EC2 instance involved here though.
I have seen a few recommendations for using PrivateLink which binds the Dynamo to VPC endpoints so that calls made from EC2 instances always go via internal network bypassing Internet.
I was wondering such a configuration is possible for lamda calling DynamoDB since lamda itself does not run in any EC2 instance and is rather serverless?
The first thing I would say is that all of your traffic between Lambda and DynamoDB is signed and encrypted, so that's typically sufficient.
There are use cases, most typically compliance reasons, when this is not sufficient. In that case you can deploy the Lambda function into a VPC of your making and configure the VPC with a private VPC endpoint for DynamoDB. Typically, the VPC would be configured without an internet gateway or NAT so that it has no egress route to the public internet. Be aware that your Lambda function startup latency will be higher than usual, because each Lambda function environment needs to attach an ENI for access to the private endpoint.
See Configuring a Lambda Function to Access Resources in an Amazon VPC.
If you don't need to access resources in a VPC, AWS recommends not to run AWS Lambda functions in a VPC. From AWS Lambda Best Practices:
Don't put your Lambda function in a VPC unless you have to. There is no benefit outside of using this to access resources you cannot expose publicly, like a private Amazon Relational Database instance. Services like Amazon Elasticsearch Service can be secured over IAM with access policies, so exposing the endpoint publicly is safe and wouldn't require you to run your function in the VPC to secure it.
Running Lambda functions in VPC adds additionally complexity, which can negatively effect scalability and performance. Each Lambda function in a VPC needs an Elastic Network Interface (ENI). Provisioning ENI's is slow and the amount of ENI's you can have is limited, so when you scale up you can run into a shortage of ENI's, preventing your Lambda functions to scale up further.
This is one way to do it.
Step 1) Deploy your lambda inside VPC.
Step 2) Create VPC Endpoint to the DynamoDB.
This should help: https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/
I need to connect dynamoDb and elasticache from aws-lambda (otherthan using NAT Gateway).
ElastiCache provides essential caching methods along with help in making the Lambda state-ful. The concern is that for Lambda to work nice with DynamoDB it should be set to NoVPC.
If we have to use ElastiCache, Lambda and both have to be in the same VPC.TO use Both ElastiCache and DynamoDB together is quite a challenge specially with Lambda. Given the VPC challenges.Do you have any suggestions to make this easier?
A Lambda function would have to have VPC access to connect to ElastiCache, and it would have to have access to resources outside the VPC to access DynamoDB so it would require a NAT gateway. There is no way to provide access to both of those services to a single Lambda function without enabling VPC access and setting up a NAT gateway.
If you just need a Redis server and aren't required specifically to use ElasiCache, then you could use a RedisLabs instance which wouldn't require you to enable VPC access on your Lambda function.
There is now a relatively easy solution for DynamoDb access from a VPC: VPC Endpoints.
"Previously, if you wanted your EC2 (elroy: or lambda) instances in your VPC to be able to access DynamoDB, you had two options. You could use an Internet Gateway (with a NAT Gateway or assigning your instances public IPs) or you could route all of your traffic to your local infrastructure via VPN or AWS Direct Connect and then back to DynamoDB."
"A VPC endpoint for DynamoDB enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB with no exposure to the public Internet...Your EC2 instances do not require public IP addresses, and you do not need an Internet gateway, a NAT device, or a virtual private gateway in your VPC. You use endpoint policies to control access to DynamoDB. Traffic between your VPC and the AWS service does not leave the Amazon network. "
The above quotes come from the links below. Note the the references to "EC2 instances" apply to lambda contexts as well.
See https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/
and
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/vpc-endpoints-dynamodb.html
I have a lambda function that accesses my Postgres db in RDS via VPC. After it queries the db, I want to post a notification to SNS. Because my lambda function exists in my VPC, it cannot access SNS. I have an internet gateway on my VPC. I read through the VPC endpoint documentation and currently only s3 is supported.
Is there anyway to publish to SNS in a lambda function in a VPC?
UPDATE
As of April 2018, SNS supports VPC Endpoints via AWS PrivateLink. So, there will be no need to set up an Internet Gateway or a NAT instance in order for a Lambda function inside your VPC to publish SNS notifications.
See this blog post for more details.
You will need a NAT server running in your VPC to route traffic outside of the VPC. AWS now offers a managed NAT service that makes this easier.
I finally managed to get it working...
The trick is that you MUST have 2 subnets.
A public one, with a routing table that sends traffic to the Internet Gateway of your VPC. Put the NAT in there.
And a private one, with a routing table that sends traffic to the NAT. Put the Lambdas in there. (BTW Making a public subnet means setting the Auto-assign Public IP option to Yes.)
It is outlined in this overview diagram from the AWS docs:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html#Configuration-2
I know this is old, but here's another option that works, for those who don't want to configure a NAT. Instead of trying to have a lambda function inside the VPC that interacts with SNS, split into 2 lambda functions, as follows.
Function 1 sits inside the VPC and interacts with the database, returning the result of your database interaction (eg, a list of IDs matching some criteria).
Function 2 sits outside the VPC, invokes Function 1, then processes the array of values and publishes the appropriate SNS notifications (eg, sends a message based on each ID in the list).
Would be nice if there was a VPC endpoint for SNS, but still in late 2016 this does not seem to be the case.