I have a lambda function that accesses my Postgres db in RDS via VPC. After it queries the db, I want to post a notification to SNS. Because my lambda function exists in my VPC, it cannot access SNS. I have an internet gateway on my VPC. I read through the VPC endpoint documentation and currently only s3 is supported.
Is there anyway to publish to SNS in a lambda function in a VPC?
UPDATE
As of April 2018, SNS supports VPC Endpoints via AWS PrivateLink. So, there will be no need to set up an Internet Gateway or a NAT instance in order for a Lambda function inside your VPC to publish SNS notifications.
See this blog post for more details.
You will need a NAT server running in your VPC to route traffic outside of the VPC. AWS now offers a managed NAT service that makes this easier.
I finally managed to get it working...
The trick is that you MUST have 2 subnets.
A public one, with a routing table that sends traffic to the Internet Gateway of your VPC. Put the NAT in there.
And a private one, with a routing table that sends traffic to the NAT. Put the Lambdas in there. (BTW Making a public subnet means setting the Auto-assign Public IP option to Yes.)
It is outlined in this overview diagram from the AWS docs:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html#Configuration-2
I know this is old, but here's another option that works, for those who don't want to configure a NAT. Instead of trying to have a lambda function inside the VPC that interacts with SNS, split into 2 lambda functions, as follows.
Function 1 sits inside the VPC and interacts with the database, returning the result of your database interaction (eg, a list of IDs matching some criteria).
Function 2 sits outside the VPC, invokes Function 1, then processes the array of values and publishes the appropriate SNS notifications (eg, sends a message based on each ID in the list).
Would be nice if there was a VPC endpoint for SNS, but still in late 2016 this does not seem to be the case.
Related
I am connecting to AWS DocumentDB from a Lambda function. In order to be able to do this I had to attach lambda to the default VPC (that's where DocumentDB cluster is running) and the default (public) subnets. But, this has caused my Lambda to timeout whenever trying make an outbound request, e.g. push message to SQS. This, I want to avoid.
So what is the recommended way of connecting to DocumentDB without loosing functionality that occurs when putting Lambda in the VPC? There's gotta be a simple solution.
Lambda functions in a VPC never get a public IP address. So if the function needs to access both VPC resources and other resources outside of the VPC the function has to be deployed only to private subnets with routes to a NAT Gateway.
Alternatively, if the only external resources you need to access are other AWS services, then you could add VPC Endpoints for those services to the VPC.
I have a lambda function sitting in a VPC, with the following in/out bound rules.
Inside the function, the goal is to trigger an email, so something like this:
const aws = require('aws-sdk');
const ses = new aws.SES({ region: 'us-west-2' });
ses.sendEmail(params, function (err, data) {
if (err) {
console.log(err);
}
});
However; when I trigger the function, there is no error printed, and the task timed out.
Originally the function was sitting out of VPC, and it can successfully send the email.
I've double checked this function's permission, which includes AWSLambdaVPCAccessExecutionRole.
Any one knows what's happening here?
the problem with this is that the lambda function's code has no path to reach the SES endpoint.
The easiest way to fix this is to give the function access to the Internet.
Although you gave the Security Group permission to connect to the internet, cannot reach the SES endpoint because it has no public IP to send the requests from.
Putting Function behind NAT
The easiest way to fix this is to:
put a NAT Gateway in the subnet(s) where the function is deployed to,
in the route table of the subnet(s) append a rule to direct all traffic to 0.0.0.0/0 to the NAT Gateway.
More info about NAT gateways
Keep in mind that this has the advantage to allow your function to access any internet resource and also the downside of routing the traffic though public Internet.
Creating an endpoint in your VPC
This solution, although cleaner and more modern, involves many steps and I suggest you to stick with the first solution.
A VPC endpoint is essentially a way to reach an AWS service (or a service from AWS's Marketplace) without letting your traffic to leave your VPC.
This works by assigning a private IP in your VPC to a "private link" to that service.
If you want to take this path, start reading from this page Interface VPC endpoints.
I did a lot of research how to solve the "Lambda won't be able to access internet/SES/S3/..." issue, because I don't want to spend money on a NAT Gateway. Here is my solution.
You need 2 Lambdas (1 in the VPC and 1 outside of the VPC)
My first Lambda does the VPC jobs (in my case request RDS)
The result of my first lambda I post to an SNS topic (You need to set VPC endpoint for SNS)
A SQS queue is subscribed to the SNS topic
My second Lambda listens to the SQS queue and does the rest of the "Internet job"
Like this I do not have to setup a NAT gateway and save those 30 bucks each month 🤘. And like this you can call SES functions like listIdentity and so on...
I have hosted a Lambda function using AWS Chalice inside a VPC since I want it to access a Serverless Aurora DB Instance. Now I also want this function to send_message() to an SQS.
I followed Tutorial: Sending a Message to an Amazon SQS Queue from Amazon Virtual Private Cloud and was able to call the SQS from inside my EC2. But even then I could not use my Lambda function to call the SQS.
It would be very helpful if someone could actually tell me how to do the whole thing manually rather than using the CloudFormation stack, or at least tell me how to get the SQS Endpoint working.
It appears that your situation is:
An Amazon VPC with an Amazon Aurora database
An AWS Lambda function that wants to communicate with the Aurora database AND an Amazon SQS queue
An AWS Lambda function can be configured as:
Connected to a subnet in a VPC, or
Not connected to a VPC, which means it is connected to the Internet
If you wish to have an AWS Lambda function communicate with resources inside a VPC AND the Internet, then you will need:
The Lambda function connected to a private subnet
A NAT Gateway in a public subnet
An Internet Gateway connected to the public subnet (it is most probably already in your VPC)
Alternatively, you can use a VPC Endpoint for SQS, which allows the Lambda function to access SQS without going to the Internet. If you are wanting to connect to multiple service (eg S3, SNS, SQS), it is probably easier just to use a NAT Gateway rather than creating VPC Endpoints for each service.
You either need to add a VPC Endpoint for SQS to your VPC, or place the Lambda function in subnets with a route to a NAT Gateway.
I have a Lambda running within a VPC which accesses secure resources (ex: RDS), but I also need it to publish an SNS notification. Is there a way to do this without having a NAT gateway?
Alternatively, I'm thinking of writing to a DynamoDB table which triggers another lambda but wanted to know if there's a simpler approach.
The simple answer is no. SNS is not currently available as a VPC endpoint so you will need to continue doing what you are already doing in order to reach RDS via lambda (NAT gateway in the private subnet).
In other words, this answer from 2016 is still relevant today -> How to let AWS lambda in a VPC to publish SNS notification?
Option A: using a NAT gateway
Your lambda is in a private subnet which means it cannot have communication with the outside world (the internet), so unless you make that a public subnet, which is of course not recommended, you cannot access the outside world. A NAT gateway allows your resources to have that access through it, and it's really not that hard to implement.
Here's a handy tutorial on how to do that:
https://github.com/naguibihab/aws-goodies/blob/master/how-to-setup-lambda-to-talk-to-internet-and-vpc.md
Option B: using a NAT instance Similar to using a NAT gateway, you can also use a NAT instance which requires a bit more administration and might have less availability but it can also be cheaper. A NAT instance works in the same way as a NAT gateway, it needs to sit in a public subnet and any lambda functions in the private subnet can access the internet through it.
As for your DynamoDB alternative, you can create a VPC Endpoint for Dynamo db as Khalid T suggested: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints-ddb.html
Edit
I have been corrected, edited my answer.
my lambda function can send message to SNS with publish method with "no VPC", but it will timeout when I put it in a VPC which has access to public internet with route and internet gateway.
Edited
I have my lambda in a public subnet which has 0.0.0.0/0 already routed to the internet gateway, so can not route it again to NAT. Is that possible to assign a EIP to lambda function
You have to add a NAT Gateway to your VPC in order for Lambda functions (and other things in your VPC that don't have a public IP) to access anything outside the VPC. You should read the Things to Know section of this AWS announcement.
Outside Internet is not accessible when VPC is enabled. So, when you send a request to publish to a SNS topic using libraries such as boto3, your requests will timeout.
NAT is one of the option but a more cost effective way would be to setup an interface endpoint to SNS from your VPN. You can go to AWS VPN console and create an endpoint and select SNS as the service.
Here is more details on how to setup your interface endpoint: https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#create-interface-endpoint