I am trying to produce the correct CDK scripts (TypeScript) to create an environment with a Lambda (accessed via API Gateway) that can call an RDS (Sql Server instance).
I believe I have it mostly working, but I wanted to connect to the RDS instance from my development machine to run some queries and check on a few things.
My RDS instance is in a private subnet, and so I believe in order to connect to it I need to add an Internet Gateway and security group to allow access on the appropriate ports.
For the life of me I can figure out the last piece, how to add the internet gateway using CDK.
The latest script I have tried is as follows:
const privateSubnectConfiguration = {
cidrMask: 26,
name: 'private-subnet',
subnetType: SubnetType.PRIVATE,
};
const publicSubnectConfiguration = {
cidrMask: 26,
name: 'public-subnet',
subnetType: SubnetType.PUBLIC,
};
const vpc = new Vpc(this, props.generateId('vpc'), {
maxAzs: 2,
subnetConfiguration: [privateSubnectConfiguration, publicSubnectConfiguration],
natGateways: 1,
});
vpc.addGatewayEndpoint(props.generateId('internet-gateway'), {
service: { name: "ig-service" }
})
Which then errors with The Vpc Endpoint Service 'ig-service' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidServiceName;
I see references to the CfnInternetGateway in the docs, but just can't figure out how to attach a new one to my VPC?
Can you please help with the syntax?
First of all, you cannot directly access a database in a private subnet. You have to deploy a proxy instance in your public subnet and forward the required ports to access your database.
When using CDK VPC construct, an Internet Gateway is created by default whenever you create a public subnet. The default route is also setup for the public subnet.
So you should remove addGatewayEndpoint() from your code, which adds a Gateway VPC Endpoint that you don't need.
You may also consider using SubnetType.ISOLATED to create a private subnet without a NAT GW, which may be redundant in your case. SubnetType.PRIVATE creates a NAT Gateway by default.
Related
I'm wanting to establish connectivity to an RDS instance from some Lambda functions. Lambda functions are autodeployed with serverless framework, so ideally my config would be dynamic. I am currently managing infrastructure with CDK, and have the following resources:
1. RDS on Private Isolated subnet in VPC A, managed by CDK
2. EC2 instance on public subnet in VPC A, managed by CDK (For access to the RDS from the wider internet)
3. (Backend) 4 Lambdas without a VPC, behind an API Gateway in default VPC, managed by serverless deploy
4. Frontend hosted on S3 behind Cloudfront, managed by serverless deploy
I can deploy the lambdas to VPC A to either the private isolated or public subnets.
Additional constraints:
Lambdas require outbound connectivity, but should be protected from inbound internet requests from public internet.
I'm a bit stumped because I don't want to update my CDK script whenever the lambdas change. Help is much appreciated.
Your lambda functions need to be in the same VPC as the database, specifically in a private subnet.
You would then adjust the security group rules to allow connectivity from the functions to the DB using something like myFynction.connections.allowToDefaultPort(myDatabaseInstance);
The VPC needs to have a NAT gateway for the lambda functions to be able to access the internet. To clarify - the functions cannot be in an isolated subnet, because isolated subnets don't have Internet connectivity. Placing the functions in a public subnet will not work either - refer to this for an explanation.
Relevant documentation: https://aws.amazon.com/premiumsupport/knowledge-center/connect-lambda-to-an-rds-instance/
I recently turned my two AWS public subnets into private subnets and added a public subnet that's got a NAT gateway. The private subnet routing table routs traffic to the NAT gateway and the public one routs it to the Internet Gateway. However, it isn't working and I don't get response to my API calls.
I think this is due to the fact that my VPC endpoint has the two private subnets associated with it instead of the public subnet. I tried to change the associated private subnets to the public one but got the AWS error:
Error modifying subnets
Can't change subnets of a requester-managed endpoint for the service ...
What would be the way to get around this error and add my public subnet to the VPC endpoint?
Additional info: Each private subnet has an EC2 auto-scaling group instance and a serverless aurora DB instance in it.
Cheers, Kris
I also had this annoying problem. The error messages are not really helpful here. They do not reveal which service exactly created those interfaces. So I went to Cloudtrail, listed all events, and searched for the VPC Endpoint name (vpce-1234567890xxx) that refused to be deleted to find out who created it. In my case, it turned out to be the RDS proxy service. So I went to RDS and deleted the proxy.
Since it is requester-managed VPC endpoint:
You cannot modify or detach a requester-managed network interface.
This means that you have to delete the resource that created the endpoint in the first place:
If you delete the resource that the network interface represents, the AWS service detaches and deletes the network interface for you.
I have been debugging, configuring and you name it the last couple of hours and i can't seem to figure out why this is happening.
I am trying to invoke a lambda function which is just retrieving basic information from ec2. when i test this lambda function in the aws console it seems to be working fine. However, invoking it in another lambda, using following code;
BasicAWSCredentials awsCreds = new BasicAWSCredentials("key1" , "key2");
AWSLambdaClientBuilder builder = AWSLambdaClientBuilder.standard()
.withRegion("eu-west-1")
.withCredentials(new AWSStaticCredentialsProvider(awsCreds));
AWSLambda client = builder.build();
InvokeRequest req = new InvokeRequest()
.withFunctionName("GetWhateverIneed");
InvokeResult result = client.invoke(req);
it simply times out. No response whatsoever... Both Lambdas are connected to the VPC and all subnets
I think it is my new VPC that is causing this problem. My VPC is consisting of:
1 VPC .
-2x Subnets (1x Private with ipv4 10.0.0.0/17, 1x Public with ipv4 10.0.128.0/17).
-1x IGW Connected to the Private subnet.
-1x NAT Gateway connected to the Public subnet .
-2x Endpoints (One for Ec2, One for SecretsManager)
I have also configured two route tables,
One for the "public" subnet:
"Routes" ->
Destination: 10.0.0.0/16 Target: local
Destination: 0.0.0.0/0 Target: My Internet Gateway(IGW)
One for the "private" subnet:
"Routes" ->
Destination: 10.0.0.0/16 Target: local .
Destination: 0.0.0.0/0 Target: My nat
I have made sure both of my Lambdas is running on the same VPC, they are both using the same security group:
This is my first time working with VPC so it is possible that i have missed something.
If your Lambda function is VPC attached, it needs to be able to communicate via your VPC to the AWS API. Lambdas do not talk to other Lambdas over the network, they initiate requests with the AWS API or an API Gateway, which passes the request on to the Lambda Function.
If you need a VPC attached Lambda to initiate another Lambda, it needs to be able to get to the AWS API or API Gateway via the internet. Alternatively, you can keep it all on private networks by adding a VPC Endpoint to the API Gateway Service.
A pattern I follow in similar circumstances is described in this previous post of mine: https://stackoverflow.com/a/43969112/6427978
i need to connect elastic cache and dynamo db from a single lambda function. My code is
exports.handler = (event, context, callback) => {
var redis = require("redis");
var client;
function connectRedisClient() {
client = redis.createClient(6379, "dgdfgdfgdfgdfgdfgfd.use1.cache.amazonaws.com", { no_ready_check: true });
}
connectRedisClient();
client.set('sampleKey', 'Hello World', redis.print);
console.log("set worked");
client.quit();
var AWS = require("aws-sdk");
var docClient = new AWS.DynamoDB.DocumentClient();
var table = "dummy";
var year = 2015;
var title = "The Big New Movie";
var params = {
TableName: table,
Item: {
"userid": "manafcj",
"year": year,
"title": title,
"test1": [645645, 7988],
"info": {
"plot": "Nothing happens at all.",
"rating": 0
}
}
};
console.log("Adding a new item...");
docClient.put(params, function (err, data) {
if (err) {
console.error("Unable to add item. Error JSON:", JSON.stringify(err, null, 2));
} else {
console.log("Added item:", JSON.stringify(data, null, 2));
}
});
callback(null, 'Hello from Lambda');
};
I executed this lambda code without configuring vpc, elastic cache section is not working , but dynamo insertion is done perfectly.
after that i made setup for VPC in my account by following steps.
create vpc
name : test-vpc-name
CIDR block:172.31.0.0/16
Tenancy:Default
Create a new subnet.
name tag : test-subnet-1a
CIDR block :172.31.0.0/20
name tag : test-subnet-1b
CIDR block :172.31.16.0/20
Create a route table
name tag : test-route-table
Create a internet gateway
name:test-internet-gateway
Attach VPC
Route all outbound 0.0.0.0/0 traffic in routes
Create a route table subnet association
Create a NAT Gateway
subnet : test-subnet-1a
also i have configured my elastic cache setup by following steps
Create subnet cache group
name : test-cache-group
Create elastic cache
type: redis
Cluster Name : test-cache
subnet cache group : test-cache-group
Finally, i have configured newly created vpc on my lambda function. Then redis-elastic cache connection is working fine, but dynamo db connection is lost. I need both working fine from a single lambda function.
I think, some fault in VPC configuration with NAT Gateway.
What is the actual issue in this setup?
Lambda and DynamoDB are executed in the AWS Public Cloud. Both are services executed in a internet facing environment. The Elastic Cache Cluster, otherwise, is user managed service that runs on your own VPC.
The first option to give access to your elastic cache cluster to your lambda function is using a NAT instance to foward external network connections to Elastic Cache cluster inside your VPC. You can get use the instructions from this document to help you with this task.
The second option, is the one that you already tried. Amazon says that when you configure this option it does not means that the Lambda will be executed inside your VPC. What is does it define the Elastic Network Interface of the Lambda container to access your VPC. At the end of day I don't think that this makes difference. You can see the details here.
But the point is, the container where your lambda is executed has only one Elastic Network Interface. If you configure your lambda to use your VPC, the Network Interface will be configured to access your subnet using a private IP and lost the internet connection. So, it will not be able to access DynamoDB unless you have a configure NAT instance/Gateway in your VPC.
As per you told us. You configured your VPC with a NAT Gateway. If all were correctly configured, this should be working. Maybe you can try the fist option, leaving your lambda outside your VPC and configuring the NAT Gateway to route the inboud connections to your Elastic Cache Cluster.
Why don't try and tell us the result?
There is now a relatively easy solution: VPC Endpoints.
"Previously, if you wanted your EC2 (elroy: or lambda) instances in your VPC to be able to access DynamoDB, you had two options. You could use an Internet Gateway (with a NAT Gateway or assigning your instances public IPs) or you could route all of your traffic to your local infrastructure via VPN or AWS Direct Connect and then back to DynamoDB."
"A VPC endpoint for DynamoDB enables Amazon EC2 instances in your VPC to use their private IP addresses to access DynamoDB with no exposure to the public Internet...Your EC2 instances do not require public IP addresses, and you do not need an Internet gateway, a NAT device, or a virtual private gateway in your VPC. You use endpoint policies to control access to DynamoDB. Traffic between your VPC and the AWS service does not leave the Amazon network. "
The above quotes come from the links below. Note the the references to "EC2 instances" apply to lambda contexts as well.
See https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/
and
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/vpc-endpoints-dynamodb.html
Edited to provide more details in line.
I'm trying to access AWS Elasticache cluster from a Lambda function using Serverless framework (v 0.5.6) without loosing access to Dynamodb. I have tried using this Gist with no luck. Inside the Lambda function, first thing I do is to connect to the Redis instance but I keep getting timeouts, I have double checked CloudFormation outputs variables and its visibility inside the function and Lambda Roles/Policies for VPC but still nothing... I haven't found either any guide on how to create VPCs and Security Groups with CloudFormation and Serverless in order to create Public and Private subnets, NATs and Internet gateways as suggested here. Can anyone help?
You will have to place the Lambda function inside the VPC that the ElastiCache cluster resides in. Of course once you do that the Lambda function only has access to resources that exist inside the VPC, so it will no longer have access to DynamoDB. The solution to that is to add a NAT gateway to the VPC, which will allow the Lambda function to access resources outside the VPC.
I would think that setting up the VPC and NAT gateway would fall outside the Serverless framework, but I'm not an expert in that framework. I would suggest looking into configuring that manually via the AWS console or doing it through something like CloudFormation, and then simply specifying in your Serverless framework configuration the VPC that it needs to use.
While it's not properly documented, you can actually configure VPC directly in the serverless config file (see link)
Version 0.5
# s-function.json
{
"name": "hello",
"runtime": "nodejs4.3",
"handler": "handler.hello”,
"endpoints": [],
"events": [],
"vpc": {
"securityGroupIds": ["sg-123456"],
"subnetIds": [
"subnet-abc1",
"subnet-abc2",
"subnet-abc3",
]
}
}
Version 1.0
# serverless.yaml
service: aws-hello
provider: aws
runtime: nodejs4.3
vpc:
securityGroupIds:
— "sg-123456"
subnetIds:
— "subnet-abc1"
— "subnet-abc1"
— "subnet-abc1"
functions:
foo: # inherits the VPC config
handler: src/handler.foo
bar: # overwrites the VPC config
handler: src/handler.bar
vpc:
securityGroupIds:
— "sg-999999"
subnetIds:
— "subnet-zzz9"
Adding summary of how I setuped this:
create a new VPC
create 3 private subnets and 2 public subnet
multiple subnets are created for redundancy
create a security group
with inbound traffic to
elastic cache port
with outbound traffic to
all ports for internet access
create a new IGW
attach this IGW with VPC
create a new NAT
select a public lambda
public connectivity type
we need 2 route tables
1 for private subnets
this will have NAT and private subnets
another for public subnet with internet access
this will have IGW and public subnets
lambda configuration
attach IAM policies to lambda functions : AWSLambdaVPCAccessExecutionRole
attach private subnets to lambda
select security group for lambda
references:
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html