When I attempt to create a VPC Endpoint for the com.amazonaws.us-east-1.lambda (lambda service), the "us-east-1a" Availability Zone is not an option. However, when I choose a different service, like "com.amazonaws.us-east-1.rds", I can choose a subnet in the "us-east-1a" Availability Zone.
I am creating VPC endpoints via CloudFormation template, but also confirmed this occurs when creating via the UI.
I have been reviewing AWS documentation and also previous questions, but I cannot determine why this is occurring and how to fix this so we can select the subnets in that AZ for that VPC endpoint. Any guidance is appreciated.
Screenshot of attempting to create VPC endpoint for lambda with us-east-1a not allowed:
Screenshot of attempting to create VPC endpoint for another service:
You can run the CLI command to check for a service and the Availability Zones which are available to use for creating a VPC endpoint.
aws ec2 describe-vpc-endpoint-services --service-names SERVICE-NAME
Example for Lambda:
aws ec2 describe-vpc-endpoint-services --service-names com.amazonaws.us-east-1.lambda
{
"ServiceDetails": [
{
"ServiceName": "com.amazonaws.us-east-1.lambda",
"AvailabilityZones": [
"us-east-1a",
"us-east-1b",
"us-east-1c"
]....}
Why can’t I select an Availability Zone for my Amazon VPC interface endpoint?
https://aws.amazon.com/premiumsupport/knowledge-center/interface-endpoint-availability-zone/
Related
Background
Testing VPC:
2 private subnets with NACLS that allow all inbound from IPs within the VPC and all outbound traffic. The subnets have a route table configured to a NAT gateway within a public subnet.
2 public subnets that allow all inbound/outbound traffic. One of the subnets contains the NAT gateway and both subnets have a route table pointing to the Internet Gateway.
Problem
When running an ECS Fargate task (platform: 1.4) within one of the private subnets, the following error arises:
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.us-west-2.amazonaws.com/: dial tcp <IP>:443: i/o timeout
The ECS task contains one container that uses a private ECR image hosted within the same AWS account. The security group associated with the task allows all inbound traffic from IPs within the VPC and allows all outbound traffic.
ECS task execution role contains the following policies:
"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
"arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
Attempts:
When the private subnets NACL's inbound access was changed to allow all types of traffic, the ECS task was strangely able to pull the ECR image.
Created the VPC endpoints mentioned in this article with the correct security groups but got the same error.
I'm tempted to try following this guide although it specifically says:
If your task definition references an image that's stored in Amazon ECR, this topic doesn't apply.
I assume you are pulling an image from ECR?
If you are launching via ecs-cli, add this to the ecs-params.yml
(read about that file here: https://github.com/aws/amazon-ecs-cli)
Firstly: auto-assign public IP addresses ->
run_params:
network_configuration:
awsvpc_configuration:
subnets:
- "subnet-06786b976xx"
- "subnet-0b9dxxxxxxx"
security_groups:
- "sg-08157xxxxxxxxx4"
assign_public_ip: "ENABLED"
Secondly make sure that the security groups you use for your VPC allow 80/443 traffic
Thirdly make sure in AWS IMA that the execution role (ecsTaskExecutionRole) has policies including:
AWSAppRunnerServicePolicyForECRAccess
Question
Why do we need VPC Endpoint for EC2?
How should it be used?
My understanding
VPC Endpoint is a service that enables services inside VPC to access outside VPC through AWS network.
For example... (See the screenshot, black thin lines)
We assume that we have a Lambda function in a private subnet and want the function to access S3.
Without VPC Endpoint: Lambda function --> NAT Gateway --> Internet Gateway -(via Internet)-> S3 bucket
With VPC Endpoint: Lambda function --> VPC Endpoint -(via AWS network)-> S3 bucket
Problem
I find VPC Endpoint for EC2 (service name: com.amazonaws.ap-southeast-1.ec2 in Singapore region). EC2 instances are always in a specific VPC, so now I do not understand why VPC Endpoint for EC2 is needed.
Does it work like: Lambda function --> VPC Endpoint -(via AWS network)-> EC2 instance (not in VPC)?
This is wrong if I understand the settings of EC2 instances correctly.
I'm trying to create a EKS cluster in a private subnet. I'm having issues getting it working. I get the error unhealthy nodes in the kubernetes cluster. Wonder if its due to security group or some other issues like VPC endpoints?
When I use NAT gateway setup then it works fine. But I don't want to use nat gateway anymore.
One think I'm not sure is should the EKS cluster subnet_ids be only private subnets?
In the below config I'm using both public and private subnets.
resource "aws_eks_cluster" "main" {
name = var.eks_cluster_name
role_arn = aws_iam_role.eks_cluster.arn
vpc_config {
subnet_ids = concat(var.public_subnet_ids, var.private_subnet_ids)
security_group_ids = [aws_security_group.eks_cluster.id, aws_security_group.eks_nodes.id, aws_security_group.external_access.id]
endpoint_private_access = true
endpoint_public_access = false
}
# Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
# Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.
depends_on = [
"aws_iam_role_policy_attachment.aws_eks_cluster_policy",
"aws_iam_role_policy_attachment.aws_eks_service_policy"
]
}
Since you don't have NAT gateway/instance, your nodes can't connect to the internet and fail as they can't "communicate with the control plane and other AWS services" (from here).
Thus, you can use VPC endpoints to enable communication with the plain and the services. To view the properly setup VPC with private subnets for EKS, you can check AWS provided VPC template for EKS (from here).
From the template, the VPC endpoints in us-east-1:
com.amazonaws.us-east-1.ec2
com.amazonaws.us-east-1.ecr.api
com.amazonaws.us-east-1.s3
com.amazonaws.us-east-1.logs
com.amazonaws.us-east-1.ecr.dkr
com.amazonaws.us-east-1.sts
Please note that all these endpoints, escept S3, are not free. So you have to consider if running cheap NAT instances or gateway would be cheaper or more expensive then maintaining these endpoints.
I am trying to restrict access to a load balancer to a fixed IP and to CodeBuild agents. To that end, I want to add a security group rule that will prevent access to the load balancer unless traffic comes from the security group associated with the CodeBuild agents (as well as a separate security group rule that restricts access except from the fixed IP). I think this means I need to associate CodeBuild with the VPC of the load balancer.
However, when I try to update an existing CodeBuild project using Terraform's codebuild_project resourcce and configure the VPC like this:
vpc_config {
security_group_ids = [var.codebuild_sg]
subnets = var.public_subnet_ids
vpc_id = var.vpc_id
}
where the variables are the security group to be used by CodeBuild, the subnets into which CodeBuild should be launched and the VPC, respectively, I get the following error message:
Error updating CodeBuild project
(arn:aws:codebuild:eu-west-2:xxxxx:project/my-project):
InvalidInputException: CodeBuild currently doesn't support VPC in
eu-west-2c, please select subnets in other availability zones.
Are there some AZs for which CodeBuild does not support VPCs? Or does this message mean something else?
Yes, CodeBuild is not available in eu-west-2c. In eu-west-2 region, CodeBuild is only available in 2 AZs (eu-west-2a and eu-west-2b).
I'm trying to access AWS Elasticache cluster from a Lambda function using Serverless framework (v 0.5.6) without loosing access to Dynamodb. I have tried using this Gist with no luck. Inside the Lambda function, first thing I do is to connect to the Redis instance but I keep getting timeouts, I have double checked CloudFormation outputs variables and its visibility inside the function and Lambda Roles/Policies for VPC but still nothing... I haven't found either any guide on how to create VPCs and Security Groups with CloudFormation and Serverless in order to create Public and Private subnets, NATs and Internet gateways as suggested here. Can anyone help?
You will have to place the Lambda function inside the VPC that the ElastiCache cluster resides in. Of course once you do that the Lambda function only has access to resources that exist inside the VPC, so it will no longer have access to DynamoDB. The solution to that is to add a NAT gateway to the VPC, which will allow the Lambda function to access resources outside the VPC.
I would think that setting up the VPC and NAT gateway would fall outside the Serverless framework, but I'm not an expert in that framework. I would suggest looking into configuring that manually via the AWS console or doing it through something like CloudFormation, and then simply specifying in your Serverless framework configuration the VPC that it needs to use.
While it's not properly documented, you can actually configure VPC directly in the serverless config file (see link)
Version 0.5
# s-function.json
{
"name": "hello",
"runtime": "nodejs4.3",
"handler": "handler.hello”,
"endpoints": [],
"events": [],
"vpc": {
"securityGroupIds": ["sg-123456"],
"subnetIds": [
"subnet-abc1",
"subnet-abc2",
"subnet-abc3",
]
}
}
Version 1.0
# serverless.yaml
service: aws-hello
provider: aws
runtime: nodejs4.3
vpc:
securityGroupIds:
— "sg-123456"
subnetIds:
— "subnet-abc1"
— "subnet-abc1"
— "subnet-abc1"
functions:
foo: # inherits the VPC config
handler: src/handler.foo
bar: # overwrites the VPC config
handler: src/handler.bar
vpc:
securityGroupIds:
— "sg-999999"
subnetIds:
— "subnet-zzz9"
Adding summary of how I setuped this:
create a new VPC
create 3 private subnets and 2 public subnet
multiple subnets are created for redundancy
create a security group
with inbound traffic to
elastic cache port
with outbound traffic to
all ports for internet access
create a new IGW
attach this IGW with VPC
create a new NAT
select a public lambda
public connectivity type
we need 2 route tables
1 for private subnets
this will have NAT and private subnets
another for public subnet with internet access
this will have IGW and public subnets
lambda configuration
attach IAM policies to lambda functions : AWSLambdaVPCAccessExecutionRole
attach private subnets to lambda
select security group for lambda
references:
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html