How can I create a target group for a network load balancer containing a VPC endpoint in Terraform?
In AWS console, I would have done following steps:
Create VPC Endpoint in two subnets to an endpoint service in another VPC
Create a target group of type IP and register the IP adresses of
the enpoints created in step 1
In terraform, I can create target groups and endpoints, but I don't know how to assign the enpoint's IPs to the target group. Where can I find instructions or an example how to do this? (Creating target groups for type instance is no problem, my question is specific for type IP).
Late to the party! But this is what I did.
Created a null resource that would get the IP addresses of the VPC endpoints and store it in a file
resource "null_resource" "nlb" {
triggers = {
always_run = "${timestamp()}"
}
provisioner "local-exec" {
command = "dig +short ${lookup(tomap(element(aws_vpc_endpoint.api-gw.dns_entry, 0)), "dns_name", "")} > /tmp/entry"
}
}
and then read the file entries
resource aws_lb_target_group_attachment nlb {
depends_on = [
null_resource.nlb
]
for_each = toset(slice(split("\n", file("/tmp/entry")), 0, 2))
target_group_arn = resource.aws_lb_target_group.nlb.arn
target_id = each.value
port = 443
}
Related
I am trying to provision EKS with node group via terraform
resource "aws_eks_node_group" "eks-node-group" {
cluster_name = aws_eks_cluster.eks-cluster.name
instance_types = var.instance_types
node_group_name = "tf-name"
node_role_arn = aws_iam_role.eks-node-group.arn
subnet_ids = var.subnet_ids
scaling_config {
desired_size = 1
max_size = 1
min_size = 1
}
update_config {
max_unavailable = 1
}
depends_on = [
aws_iam_role_policy_attachment.eks-node-group-worker-node-policy,
aws_iam_role_policy_attachment.eks-node-group-cni-policy,
aws_iam_role_policy_attachment.eks-node-group-registry-read-only-policy
]
}
I am trying to provision it using private subnet.
However I am getting an error of
One or more Amazon EC2 Subnets of [] for node group does not
automatically assign public IP addresses to instances launched into
it. If you want your instances to be assigned a public IP address,
then you need to enable auto-assign public IP address for the subnet.
See IP addressing in VPC guide:
What do I need to do?
You can find your answer from AWS Documentation : Managed node groups
"Amazon EKS managed node groups can be launched in both public and private subnets. If you launch a managed node group in a public subnet on or after April 22, 2020, the subnet must have MapPublicIpOnLaunch set to true for the instances to successfully join a cluster. If the public subnet was created using eksctl or the Amazon EKS vended AWS CloudFormation templates on or after March 26, 2020, then this setting is already set to true. If the public subnets were created before March 26, 2020, you must change the setting manually. For more information, see Modifying the public IPv4 addressing attribute for your subnet."
I am trying to setup AWS SFTP transfer in vpc endpoint mode but there is one think I can't manage with.
The problem I have is how to get target IPs for NLB target group.
The only output I found:
output "vpc_endpoint_transferserver_network_interface_ids" {
description = "One or more network interfaces for the VPC Endpoint for transferserver"
value = flatten(aws_vpc_endpoint.transfer_server.*.network_interface_ids)
}
gives network interface ids which cannot be used as targets:
Outputs:
api_url = https://12345.execute-api.eu-west-1.amazonaws.com/prod
vpc_endpoint_transferserver_network_interface_ids = [
"eni-12345",
"eni-67890",
"eni-abcde",
]
I went through:
terraform get subnet integration ips from vpc endpoint subnets tab
and
Terraform how to get IP address of aws_lb
but none of them seems to be working. The latter says:
on modules/sftp/main.tf line 134, in data "aws_network_interface" "ifs":
134: count = "${length(local.nlb_interface_ids)}"
The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.
You can create an Elastic IP
resource "aws_eip" "lb" {
instance = "${aws_instance.web.id}"
vpc = true
}
Then specify the Elastic IPs while creating Network LB
resource "aws_lb" "example" {
name = "example"
load_balancer_type = "network"
subnet_mapping {
subnet_id = "${aws_subnet.example1.id}"
allocation_id = "${aws_eip.example1.id}"
}
subnet_mapping {
subnet_id = "${aws_subnet.example2.id}"
allocation_id = "${aws_eip.example2.id}"
}
}
Question
What/where is the definition of AWS Prefix?
Background
While looking for a way to list S3 endpoint CIDR, encountered the word AWS prefix list but not sure what it exactly means and where the terminology is defined.
Confusion
Prefix means a word placed in front. For S3, according to Listing Keys Hierarchically Using a Prefix and Delimiter, it should be the starting path to an object.
However, apparently it refers to a IP address range. How come prefix is used for IP ranges? What is the history or reason?
Terraform aws_prefix_list
This can be used both to validate a prefix list given in a variable and to obtain the CIDR blocks (IP address ranges) for the associated AWS service.
describe-prefix-lists
Describes available AWS services in a prefix list format, which includes the prefix list name and prefix list ID of the service and the IP address range for the service.
AWS IP Address Ranges
SERVICE="S3"
REGION="us-west-1"
$ curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | \
jq -r --arg SERVICE "$SERVICE" --arg REGION "${REGION}" '.prefixes[] \
| select(.service==$SERVICE and .region==$REGION)'
{
"ip_prefix": "52.92.48.0/22",
"region": "us-west-1",
"service": "S3"
}
{
"ip_prefix": "54.231.232.0/21",
"region": "us-west-1",
"service": "S3"
}
{
"ip_prefix": "52.219.20.0/22",
"region": "us-west-1",
"service": "S3"
}
{
"ip_prefix": "52.219.24.0/21",
"region": "us-west-1",
"service": "S3"
}
Update
Gateway VPC Endpoints
Specify the VPC in which to create the endpoint, and the service to which you're connecting. A service is identified by a prefix list—the name and ID of a service for a Region. A prefix list ID uses the form pl-xxxxxxx and a prefix list name uses the form "com.amazonaws.region.service". Use the prefix list name (service name) to create an endpoint.
what is the meaning of Prefix ?
suppose you have a network like 10.5.10.0/24
so you will have the 10.5.10 prefix in that subnet from 1 to 255 and your network address will be 10.5.10.0
I suppose (10.0.0.0/24) means (Top 24 bit part of 32 bit IP) of a network that has
254 ip addresses from 1 to 254 (0 is network and 255 is broadcast). Prefix is top 24 bit and suffix (?) is last 8 bit. List of top N bit which identifies a network is a list of IP prefix.
If what you are looking for is prefix list id for vpc endpoint like dynamodb/s3, then it is not related to IP or CIDR. As it is mentioned in the documentation:
A prefix list ID is required for creating an outbound security group rule that allows traffic from a VPC to access an AWS service through a gateway VPC endpoint.
So if do not have prefix-list id in your security group outbout for ec2 or vpc-lambda, you will get time out when connecting to dynamodb or s3.
You can get the prefix-list by running
aws ec2 describe-prefix-lists
{
"PrefixLists": [
{
"Cidrs": [
"54.231.0.0/17",
"52.216.0.0/15"
],
"PrefixListId": "pl-63c5400k",
"PrefixListName": "com.amazonaws.us-east-1.s3"
},
{
"Cidrs": [
"52.94.0.0/22",
"52.119.224.0/20"
],
"PrefixListId": "pl-02ad2a6c",
"PrefixListName": "com.amazonaws.us-east-1.dynamodb"
}
]
}
Then you can put this PrefixListId into your security group outbound via aws web console. If you use terraform for different region, it could be something like:
resource "aws_security_group_rule" "MyService_to_DynamoDB_east" {
count = "${ lower(var.region) == "us-east-1" ? 1 : 0 }"
security_group_id = "${aws_security_group.MyService_Ext_Api.id}"
description = "DynamoDB"
type = "egress"
protocol = "tcp"
from_port = 443
to_port = 443
prefix_list_ids = ["pl-02ad2a6c"]
}
resource "aws_security_group_rule" "MyService_to_DynamoDB_west" {
count = "${ lower(var.region) == "us-west-2" ? 1 : 0 }"
security_group_id = "${aws_security_group.MyService_Ext_Api.id}"
description = "DynamoDB"
type = "egress"
protocol = "tcp"
from_port = 443
to_port = 443
prefix_list_ids = ["pl-0ca54061"]
}
The term prefix list comes from routing technology. An IP address in CIDR format has an IP prefix and a network prefix (10.1.0.0/16). The IP prefix is 10.1 and the network prefix is /16.
Therefore if you are using a list of IP addresses in CIDR format we call it an IP Prefix List.
What is Prefix List in AWS VPC context
Prefix List is the IP ranges in CIDR format which are assigned to the VPC Gateway Endpoint (S3 or DynamoDB) in a region. It is region-specific.
For example, as of 24/Feb, 2020, the prefix list for DynamoDB in the us-east-2 region is "52.94.4.0/24" where 52.94.4 is the IP prefix and the network prefix is /24 as explained by #John Hanley. The IPs the VPC Gateway endpoint for DynamoDB can take is within 52.94.4.1 - 52.94.4.254 (AWS may have reserved some IPs).
$ aws ec2 describe-prefix-lists
{
"PrefixLists": [
{
"Cidrs": [
"52.94.4.0/24"
],
"PrefixListId": "pl-4ca54025",
"PrefixListName": "com.amazonaws.us-east-2.dynamodb"
},
{
"Cidrs": [
"52.219.80.0/20",
"3.5.128.0/22",
"3.5.132.0/23",
"52.219.96.0/20",
"52.92.76.0/22"
],
"PrefixListId": "pl-7ba54012",
"PrefixListName": "com.amazonaws.us-east-2.s3"
}
]
}
Those prefix list has an ID and a name. We can specify prefix list ID in a VPC routing table and in a Security Group, but not in NACL. We need to use CIDR for NACL.
Gateway VPC Endpoints
You canNOT use a prefix list ID in an outbound rule in a network ACL to allow or deny outbound traffic to the service specified in an endpoint. If your network ACL rules restrict traffic, you must specify the CIDR block (IP address range) for the service instead. You can, however, use a prefix list ID in an outbound security group rule. For more information, see Security Groups.
Terraform examples
NACL
Data Source: aws_prefix_list
resource "aws_vpc_endpoint" "private_s3" {
vpc_id = "${aws_vpc.foo.id}"
service_name = "com.amazonaws.us-west-2.s3"
}
data "aws_prefix_list" "private_s3" {
prefix_list_id = "${aws_vpc_endpoint.private_s3.prefix_list_id}"
}
resource "aws_network_acl" "bar" {
vpc_id = "${aws_vpc.foo.id}"
}
resource "aws_network_acl_rule" "private_s3" {
network_acl_id = "${aws_network_acl.bar.id}"
rule_number = 200
egress = false
protocol = "tcp"
rule_action = "allow"
cidr_block = "${data.aws_prefix_list.private_s3.cidr_blocks[0]}"
from_port = 443
to_port = 443
}
Security Group as provided by #LeOn - Han Li in his answer.
resource "aws_security_group_rule" "MyService_to_DynamoDB_east" {
count = "${ lower(var.region) == "us-east-1" ? 1 : 0 }"
security_group_id = "${aws_security_group.MyService_Ext_Api.id}"
description = "DynamoDB"
type = "egress"
protocol = "tcp"
from_port = 443
to_port = 443
prefix_list_ids = ["pl-02ad2a6c"]
}
resource "aws_security_group_rule" "MyService_to_DynamoDB_west" {
count = "${ lower(var.region) == "us-west-2" ? 1 : 0 }"
security_group_id = "${aws_security_group.MyService_Ext_Api.id}"
description = "DynamoDB"
type = "egress"
protocol = "tcp"
from_port = 443
to_port = 443
prefix_list_ids = ["pl-0ca54061"]
}
Access Control
Why can’t I connect to an S3 bucket using a gateway VPC endpoint? layouts comprehensive list regarding S3 VPC Gateway Endpoint and the prefix list ID and CIDR would play parts there.
DNS settings in your VPC
Important: DNS resolution must be enabled in your VPC (see Gateway Endpoint Limitations). If you're using your own DNS server, be sure DNS requests to AWS services resolve to AWS-maintained IP addresses.
Route table settings to Amazon S3
Security group outbound rules
Network ACL rules
Gateway VPC endpoint policy
S3 bucket policy
IAM policy
References
Gateway VPC Endpoints
Specify the VPC in which to create the endpoint, and the service to which you're connecting. A service is identified by a prefix list—the name and ID of a service for a Region. A prefix list ID uses the form pl-xxxxxxx and a prefix list name uses the form "com.amazonaws.region.service". Use the prefix list name (service name) to create an endpoint.
Well even though question is answered regarding Prefix definition pertaining to VPC, I did not see answer to prefix definition related to S3 (beside CIDR related). Simply put in S3 if you have a folder Photos and it has yourpic.jpg file in it then it should look like this photos/yourpic.jpg. The part which is folder photos/ is called prefix in S3 Terminology. As mentioned in latest AWS Performance tuning guide for S3 we can use prefix to do parallelism and increase performance. Anyway I am still learning AWS so take it with a grain of salt. Check below link for reference.
https://docs.aws.amazon.com/AmazonS3/latest/user-guide/using-folders.html
An AWS-managed prefix list is a way to add a single rule in an AWS Security Group that represents all the IP ranges for that service instead of having to enter every single range in the AWS IP ranges here:
https://ip-ranges.amazonaws.com/ip-ranges.json
For example, if you want to provide outbound access to S3 you would have to enter every IP range in that JSON IP ranges list to access S3 (or the ones in the region you're in if you are restricting access to a single region.)
With a prefix list, you create a rule in the security group and assign the prefix list instead as the destination.
You can also create your own customer-managed prefix list for your own list of IP ranges. I just wrote a blog post on how to do that on my medium Cloud-Security blog.
I followed "https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html" to create an EKS cluster using terraform.
I was able to create a config map successfully but i am unable to get the node details -
$ ./kubectl_1.10.3_darwin get nodes
No resources found.
Service details -
$ ./kubectl_1.10.3_darwin get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 2h
Kubectl logs on nodes -
Aug 5 09:14:32 ip-172-31-18-205 kubelet: I0805 09:14:32.617738 25463 aws.go:1026] Building AWS cloudprovider
Aug 5 09:14:32 ip-172-31-18-205 kubelet: I0805 09:14:32.618168 25463 aws.go:988] Zone not specified in configuration file; querying AWS metadata service
Aug 5 09:14:32 ip-172-31-18-205 kubelet: E0805 09:14:32.794914 25463 tags.go:94] Tag "KubernetesCluster" nor "kubernetes.io/cluster/..." not found; Kubernetes may behave unexpectedly.
Aug 5 09:14:32 ip-172-31-18-205 kubelet: F0805 09:14:32.795622 25463 server.go:233] failed to run Kubelet: could not init cloud provider "aws": AWS cloud failed to find ClusterID
Aug 5 09:14:32 ip-172-31-18-205 systemd: kubelet.service: main process exited, code=exited, status=255/n/a
Aug 5 09:14:32 ip-172-31-18-205 systemd: Unit kubelet.service entered failed state.
Aug 5 09:14:32 ip-172-31-18-205 systemd: kubelet.service failed.
AWS getting started documentation doesn't mention any tags related information "https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html".
After a while I found out that I missed to put resource tags like "kubernetes.io/cluster/*" to my networking resources.
My networking resources are pre-created, I use remote states to fetch the required details. I believe that I can either add tags to it OR create a new VPC env.
Is there any alternate way to solve this without adding tags or provisioning new resources?
Make sure you add a similar tag as below to your VPCs, Subnets & ASGs -
"kubernetes.io/cluster/${CLUSTER_NAME}" = "shared"
NOTE: The usage of the specific kubernetes.io/cluster/* resource tags below are required for EKS and Kubernetes to discover and manage networking resources.
NOTE: The usage of the specific kubernetes.io/cluster/* resource tag below is required for EKS and Kubernetes to discover and manage compute resources. - Terraform docs
I had missed propagating tags using auto-scaling groups on worker nodes. I added below code to ASG terraform module & it started working, at least the nodes were able to connect to the master cluster. You also need to add the tag to VPC & Subnets for EKS and Kubernetes to discover and manage networking resources.
For VPC -
locals {
cluster_tags = {
"kubernetes.io/cluster/${var.project}-${var.env}-cluster" = "shared"
}
}
resource "aws_vpc" "myvpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
tags = "${merge(map("Name", format("%s-%s-vpcs", var.project, var.env)), var.default_tags, var.cluster_tags)}"
}
resource "aws_subnet" "private_subnet" {
count = "${length(var.private_subnets)}"
vpc_id = "${aws_vpc.myvpc.id}"
cidr_block = "${var.private_subnets[count.index]}"
availability_zone = "${element(var.azs, count.index)}"
tags = "${merge(map("Name", format("%s-%s-pvt-%s", var.project, var.env, element(var.azs, count.index))), var.default_tags, var.cluster_tags)}"
}
resource "aws_subnet" "public_subnet" {
count = "${length(var.public_subnets)}"
vpc_id = "${aws_vpc.myvpc.id}"
cidr_block = "${var.public_subnets[count.index]}"
availability_zone = "${element(var.azs, count.index)}"
map_public_ip_on_launch = "true"
tags = "${merge(map("Name", format("%s-%s-pub-%s", var.project, var.env, element(var.azs, count.index))), var.default_tags, var.cluster_tags)}"
}
For ASGs -
resource "aws_autoscaling_group" "asg-node" {
name = "${var.project}-${var.env}-asg-${aws_launch_configuration.lc-node.name}"
vpc_zone_identifier = ["${var.vpc_zone_identifier}"]
min_size = 1
desired_capacity = 1
max_size = 1
target_group_arns = ["${var.target_group_arns}"]
default_cooldown= 100
health_check_grace_period = 100
termination_policies = ["ClosestToNextInstanceHour", "NewestInstance"]
health_check_type="EC2"
depends_on = ["aws_launch_configuration.lc-node"]
launch_configuration = "${aws_launch_configuration.lc-node.name}"
lifecycle {
create_before_destroy = true
}
tags = ["${data.null_data_source.tags.*.outputs}"]
tags = [
{
key = "Name"
value = "${var.project}-${var.env}-asg-eks"
propagate_at_launch = true
},
{
key = "role"
value = "eks-worker"
propagate_at_launch = true
},
{
key = "kubernetes.io/cluster/${var.project}-${var.env}-cluster"
value = "owned"
propagate_at_launch = true
}
]
}
I was able to deploy a sample application post above changes.
PS - Answering this since AWS EKS getting started documentation doesn't have these instructions very clear & people trying to create ASGs manually may fall into this issue. This might help others save their time.
I tried to summarize below all the resources that requires tagging - I hope I haven't missed something.
Tagging Network resources
(Summary of this doc).
1) VPC tagging requirement
When you create an Amazon EKS cluster earlier than version 1.15, Amazon EKS tags the VPC containing the subnets you specify in the following way so that Kubernetes can discover it:
Key Value
kubernetes.io/cluster/<cluster-name> shared
Key: The value matches your Amazon EKS cluster's name.
Value: The shared value allows more than one cluster to use this VPC.
2) Subnet tagging requirement
When you create your Amazon EKS cluster, Amazon EKS tags the subnets you specify in the following way so that Kubernetes can discover them:
Note: All subnets (public and private) that your cluster uses for
resources should have this tag.
Key Value
kubernetes.io/cluster/<cluster-name> shared
Key: The value matches your Amazon EKS cluster.
Value: The shared value allows more than one cluster to use this subnet.
3) Private subnet tagging requirement for internal load balancers
Private subnets must be tagged in the following way so that Kubernetes knows it can use the subnets for internal load balancers. If you use an Amazon EKS AWS CloudFormation template to create...
Key Value
kubernetes.io/role/internal-elb 1
4) Public subnet tagging option for external load balancers
You must tag the public subnets in your VPC so that Kubernetes knows to use only those subnets for external load balancers instead of choosing a public subnet in each Availability Zone (in lexicographical order by subnet ID). If you use an Amazon EKS AWS CloudFormation template...
Key Value
kubernetes.io/role/elb 1
Tagging Auto Scaling group
(Summary of this doc).
The Cluster Autoscaler requires the following tags on your node group Auto Scaling groups so that they can be auto-discovered.
If you used the previous eksctl commands to create your node groups, these tags are automatically applied. If not, you must manually tag your Auto Scaling groups with the following tags.
Key Value
k8s.io/cluster-autoscaler/<cluster-name> owned
k8s.io/cluster-autoscaler/enabled true
Tagging Security groups
(Taken from the end of this doc).
If you have more than one security group associated to your nodes, then one of the security groups must have the following tag applied to it. If you have only one security group associated to your nodes, then the tag is optional.
Key Value
kubernetes.io/cluster/<cluster-name> owned
I am trying to create an interface type kms endpoint in terraform on aws. While doing so, I get the following error -
Error creating VPC Endpoint: InvalidParameter: Subnet: subnet- does not have corresponding zone in the service com.amazonaws.ap-southeast-1.kms
My endpoint resource looks like --
resource "aws_vpc_endpoint" "kms" {
vpc_id = "${aws_vpc.main.id}"
service_name = "com.amazonaws.${var.aws_region}.kms"
vpc_endpoint_type = "Interface"
subnet_ids = [
<my subnet ids>
]
security_group_ids = [ <my sg ids> ]
private_dns_enabled = true
}
Anyone any clue on what I might be missing. Just FYI -- I haven't added any route53 record for kms. Do i need to?
Looking forward to any replies.
Thanks
Avi
If you have not created the private DNS in Route53, set private_dns_enabled=false . Otherwise create the private zone first.
I solved this issue by creating multiple endpoint resources for different subnets.