How Do I Use A Terraform Data Source To Reference A Managed Prefix List? - amazon-web-services

I'm trying to update a terraform module to add a new security group, which will have an inbound rule populated with two managed prefix lists. The prefix lists are shared to my AWS account from a different account using AWS Resource Access Manager, however I have tried referencing prefix lists created within my own AWS account and am seeing the same error.
Below is the terraform I am using:
resource "aws_security_group" "akamai_sg" {
name = "akamai-pl-sg"
description = "Manage access from Akamai to ${var.environment} alb"
vpc_id = var.vpc_id
tags = merge(var.common_tags, tomap({ "Name" = "akamai-pl-sg" }))
revoke_rules_on_delete = true
}
resource "aws_security_group_rule" "akamai_to_internal_alb" {
for_each = toset(var.domains_inc_akamai)
type = "ingress"
description = "Allow Akamai into ${var.environment}${var.domain_name_suffix}-alb"
from_port = var.alb_listener_port
to_port = var.alb_listener_port
protocol = "tcp"
security_group_id = aws_security_group.akamai_sg.id
prefix_list_ids = [data.aws_prefix_list.akamai-site-shield.id, data.aws_prefix_list.akamai-staging.id]
}
data "aws_prefix_list" "akamai-site-shield" {
filter {
name = "prefix-list-id"
values = ["pl-xxxxxxxxxx"]
}
}
data "aws_prefix_list" "akamai-staging" {
filter {
name = "prefix-list-id"
values = ["pl-xxxxxxxxxx"]
}
}
The terraform error I am revieving reads:
"Error: no matching prefix list found; the prefix list ID or name may be invalid or not exist in the current region"
Is anyone able to help, or see where I am going wrong?
Thanks in advance.

Would not be the following possible?
data "aws_vpc_endpoint" "s3" {
vpc_id = aws_vpc.foo.id
service_name = "com.amazonaws.us-west-2.s3"
}
data "aws_prefix_list" "s3" {
prefix_list_id = aws_vpc_endpoint.s3.prefix_list_id
}
It seems the solution is to use:
data "aws_ec2_managed_prefix_list" "example" {
filter {
name = "prefix-list-name"
values = ["my-prefix-list"]
}
}

Related

no matching EC2 Security Group found

I create modules for creating AWS vpc, security group and EC2. I make EC2 module depend on vpc and the security group module.
but I get that error:
no matching EC2 Security Group found
on .terraform/modules/main_ec2/modules/EC2Module/main.tf line 26, in data "aws_security_group" "security_group_id":
26: data "aws_security_group" "security_group_id" {
EC2 module
data "aws_subnet" "subnet_id" {
vpc_id = var.vpc_id
count = length(var.subnet_name)
depends_on = [var.subnet_id_depends_on]
filter {
name = "tag:Name"
values = ["VPC0${var.vpc_number}-${element(var.subnet_name, count.index)}"]
}
}
data "aws_security_group" "security_group_id" {
count = length(var.Server_Group_Name)
depends_on = [var.security_group_depends_on]
filter {
name = "tag:Name"
values = ["${local.name}-SG-${var.Server_Group_Name[count.index]}-${format("%02d", count.index + 1)}"]
}
}
resource "aws_instance" "create_instance" {
ami = "${element(data.aws_ami_ids.ami_id.*.id, count.index)}"
instance_type = var.EC2_Type[count.index]
subnet_id = "${element(data.aws_subnet.subnet_id.*.id, count.index)}"
associate_public_ip_address = "true"
vpc_security_group_ids = [ data.aws_security_group.security_group_id[count.index].id ]
key_name = "${local.name}-KP-${var.Server_Name[count.index]}"
depends_on = [var.EC2_depends_on]
}
sg module
resource "aws_security_group" "SGS" {
count = length(var.Server_Group_Name)
name = "${local.name}-SG-${var.Server_Group_Name[count.index]}-${format("%02d", count.index + 1)}"
description = "${local.name}-SG-${var.Server_Group_Name[count.index]}-${format("%02d", count.index + 1)}"
vpc_id = var.vpc_id
tags = {
name = "tag:Name"
values = "${local.name}-SG-${var.Server_Group_Name[count.index]}-${format("%02d", count.index + 1)}"
}
varibles.tf
variable "subnet_id_depends_on" {
type = list(string)
default = ["module.vpc_main"]
}
variable "security_group_depends_on" {
type = list(string)
default = ["module.sg"]
}
variable "EC2_depends_on" {
type = list(string)
default = [ "module.vpc_main", "module.sg", "module.key_pair" ]
}
for example, I call modules like that:
module "main_ec2" {
source = "/home/reham/Data/projectes/terraform//modules/EC2Module"
subnet_id_depends_on = var.subnet_id_depends_on
security_group_depends_on = var.security_group_depends_on
ami_name = var.ami_name
EC2_depends_on = var.EC2_depends_on
}
Outputs defined in the sg module:
output "vpc_id" {
description = "VPC ID"
value = aws_vpc.vpc.id
}
output "primary_public_subnets_id" {
description = "public subnet ID"
value = aws_subnet.primary_public_subnets.*.id
}
output "secondary_public_subnets_id" {
description = "public subnet ID"
value = aws_subnet.secondary_public_subnets.*.id
}
output "primary_private_subnets_id" {
description = "public subnet ID"
value = aws_subnet.primary_private_subnets.*.id
}
output "security-groups" {
value = aws_security_group.SGS[*].id
}
what must I do to solve this problem?
There is more than one issue with the code and it is a result of probably not understanding how the module outputs work [1]. The second part of the issue is that there are many explicit dependencies defined with depends_on which are not required to be there since implicit dependencies are good enough [2]. Furthermore, as per comments, data sources in the EC2 module are not required as information about VPC, subnets and security groups can be provided with the outputs. Additionally, the EC2 instance resource is using count, but there is no count meta-argument there, so it will not work even if all the inputs are fixed. So here are my suggestions:
Redefine variables in the EC2 module (no need for depdends_on) and remove data sources (except possibly the one for the AMI)
Decide which variable you are going to use for the count meta-argument in the EC2 instance resource
I am going to give you an example with one subnet and security groups and then you can build from there. The EC2 module fixes:
resource "aws_instance" "create_instance" {
ami = data.aws_ami_ids.ami_id[0].id
instance_type = var.EC2_Type
subnet_id = var.subnet_id
associate_public_ip_address = "true"
vpc_security_group_ids = var.security_group_ids
key_name = "${local.name}-KP-${var.Server_Name[0]}"
}
In EC2 module, the variables would have to be changed to:
variable "subnet_id" {}
variable "security_group_ids" {}
Then, in the root module:
module "main_ec2" {
source = "/home/reham/Data/projectes/terraform//modules/EC2Module"
subnet_id = module.sg.primary_public_subnets_id[0]
security_group_ids = module.sg.security-groups
ami_name = var.ami_name
}
Referencing the module outputs in the EC2 module will make sure that the SG module is created first (that is an implicit dependency). At best this should give you a result for one EC2 instance as it seems there are many resources that rely on using the count meta-argument and that code is not in the question.
[1] https://www.terraform.io/language/values/outputs#accessing-child-module-outputs
[2] https://www.terraform.io/language/resources/behavior#resource-dependencies

Completely private certificate with AWS in Terraform?

I would like to create a certificate signed by AWS for use by internal services. The internal services are only visible inside my VPC. I don't want anything about the internal services, such as the subdomain, to leak externally.
This is the bit of Terraform I am unsure about:
resource "aws_acm_certificate" "internal" {
domain_name = "*.internal.example.org."
# What goes here?
}
The internal service consumes the certificate like this:
resource "aws_elastic_beanstalk_environment" "foo" {
name = "foo-env"
# ...
setting {
namespace = "aws:ec2:vpc"
name = "ELBScheme"
value = "internal"
resource = ""
}
setting {
namespace = "aws:elbv2:listener:443"
name = "SSLCertificateArns"
value = aws_acm_certificate.internal.arn
resource = ""
}
}
I then assign an internal DNS entry like this:
resource "aws_route53_zone" "private" {
name = "example.org."
vpc {
vpc_id = aws_vpc.main.id
}
}
resource "aws_route53_record" "private_cname_foo" {
zone_id = aws_route53_zone.private.zone_id
name = "foo.internal.example.org."
type = "CNAME"
ttl = "300"
records = [
aws_elastic_beanstalk_environment.foo.cname
]
}
How do I get AWS to create a certificate for me?

How to add the `default-allow-http`

How to add the default-allow-http firewall rule in a terraform script to a Google Cloud Compute Instance?
provider "google" {
credentials = file("CREDENTIAL_FILE")
project = "gitlab-project"
region = var.region
}
resource "google_compute_instance" "gitlab" {
name = var.machine_specs.name
machine_type = var.machine_type.emicro
zone = var.zone
boot_disk {
initialize_params {
image = var.machine_specs.os
size = var.machine_specs.size
}
}
network_interface {
# A default network is created for all GCP projects
network = "default"
access_config {
nat_ip = google_compute_address.static.address
}
}
// Add the SSH key
metadata = {
ssh-keys = "martin:${file("~/.ssh/id_rsa.pub")}"
}
}
// A variable for extracting the external ip of the instance
output "ip" {
value = "${google_compute_instance.gitlab.network_interface.0.access_config.0.nat_ip}"
}
resource "google_compute_address" "static" {
name = "ipv4-address"
address_type = "EXTERNAL"
address = "XXX.XXX.XXX.XXX"
}
resource "google_compute_firewall" "allow-http" {
name = "default-allow-http"
network =
allow{
protocol = "tcp"
ports = ["80"]
}
}
You can use tags argument available in google_compute_instance resource.
it would look something like:
resource "google_compute_instance" "gitlab" {
name = var.machine_specs.name
machine_type = var.machine_type.emicro
zone = var.zone
tags = ["http-server"]
http-server tag is for default-allow-http firewall rule.
If you need default-allow-https then simply append https-server to the tag list.
Hope this helps.
You need to add the tags ["http-server", "https-server"] to your resource group google_compute_instance like so:
[...]
resource "google_compute_instance" "gitlab" {
name = var.machine_specs.name
machine_type = var.machine_type.emicro
zone = var.zone
tags = ["http-server", "https-server"]
[...]
Simply add the tags http-server and https-server to your google_cloud_instance resource gruop.
The tags can be found in the Firewall-settings in your GCloud-Console.

How to load balance google compute instance using terraform?

In my terraform configuration file, I define my resource like so:
resource "google_compute_instance" "test" {
...
count = 2
}
What I now want is to create load balancer, that will balance between two instances of my google compute instance. Unfortunatelly, I could not find in documentation anything relative to this task. It seems like google_compute_target_pool or google_compute_lb_ip_ranges have nothing to do with my problem.
You would have to use 'forwarding rules' as indicated on this terraform document. To use load balancing and protocol forwarding, you must create a forwarding rule that directs traffic to specific target instances. The use on Cloud Platform of forwarding rules you can find here.
In common cases you can use something like the following:
resource "google_compute_instance" "test" {
name = "nlb-node${count.index}"
zone = "europe-west3-b"
machine_type = "f1-micro"
count = 2
boot_disk {
auto_delete = true
initialize_params {
image = "ubuntu-os-cloud/ubuntu-1604-lts"
size = 10
type = "pd-ssd"
}
}
network_interface {
subnetwork = "default"
access_config {
nat_ip = ""
}
}
service_account {
scopes = ["userinfo-email", "compute-ro", "storage-ro"]
}
}
resource "google_compute_http_health_check" "nlb-hc" {
name = "nlb-health-checks"
request_path = "/"
port = 80
check_interval_sec = 10
timeout_sec = 3
}
resource "google_compute_target_pool" "nlb-target-pool" {
name = "nlb-target-pool"
session_affinity = "NONE"
region = "europe-west3"
instances = [
"${google_compute_instance.test.*.self_link}"
]
health_checks = [
"${google_compute_http_health_check.nlb-hc.name}"
]
}
resource "google_compute_forwarding_rule" "network-load-balancer" {
name = "nlb-test"
region = "europe-west3"
target = "${google_compute_target_pool.nlb-target-pool.self_link}"
port_range = "80"
ip_protocol = "TCP"
load_balancing_scheme = "EXTERNAL"
}
You can get load balancer external ip via ${google_compute_forwarding_rule.network-load-balancer.ip_address}
// output.tf
output "network_load_balancer_ip" {
value = "${google_compute_forwarding_rule.network-load-balancer.ip_address}"
}

Not able to add multiple target_id inside targer group using terraform

I'm trying to create target groups and attach multiple machines to the target groups using terraform script.
I'm not able to attach multiple target_id please help me to achieve this.
As of Terraform 0.12, this could simply be
resource "aws_alb_target_group_attachment" "test" {
count = length(aws_instance.test)
target_group_arn = aws_alb_target_group.test.arn
target_id = aws_instance.test[count.index].id
}
assuming aws_instance.test returns a list.
https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9 is an excellent reference.
Below code actually works for me.
resource "aws_alb_target_group_attachment" "test" {
count = 3 #This can be passed as variable.
target_group_arn = "${aws_alb_target_group.test.arn}"
target_id = "${element(split(",", join(",", aws_instance.web.*.id)), count.index)}"
}
Ref:
https://github.com/terraform-providers/terraform-provider-aws/issues/357
https://groups.google.com/forum/#!msg/terraform-tool/Mr7F3W8WZdk/ouVR3YsrAQAJ
Thanks for your quick reply.
Actually giving seperate tag like test1 and test2 for aws_alb_target_group_attachment helped me to add multiple target instances inside one taget group.
resource "aws_alb_target_group_attachment" "test1" {
target_group_arn = "${aws_alb_target_group.test.arn}"
port = 8080
target_id = "${aws_instance.inst1.id}"
}
resource "aws_alb_target_group_attachment" "test2" {
target_group_arn = "${aws_alb_target_group.test.arn}"
port = 8080
target_id = "${aws_instance.inst2.id}"
}
Try creating a list of instance ID's and then iterate over using the count index.
For example:
variable "instance_list" {
description = "Push these instances to ALB"
type = "list"
default = ["i00001", "i00002", "i00003"]
}
resource "aws_alb_target_group_attachment" "test" {
count = "${var.instance_list}"
target_group_arn = "${aws_alb_target_group.test.arn}"
target_id = "${element(var.instance_list, count.index)}"
port = 80
}
I created an EMR from terraform and attached multiple "CORE" type EC2 instances to a target group.
The first step would be to retrieve existing instances (which are in "running" state)
data "aws_instances" "core_instances" {
instance_state_names = ["running"]
instance_tags = {
"aws:elasticmapreduce:instance-group-role" = "CORE"
"terraform" = "true"
}
}
Next, retrieve an existing VPC
data "aws_vpc" "test_vpc" {
filter {
name = "tag:Name"
values = ["your-vpc-name"]
}
}
Use the above data to create a target group and then attach instances to it:
resource "aws_lb_target_group" "core_lb" {
name = "core-emr-target-group"
port = 8765
protocol = "TCP"
target_type = "instance"
vpc_id = data.aws_vpc.test_vpc.id
}
resource "aws_lb_target_group_attachment" "core_lb_instances" {
for_each = toset(data.aws_instances.core_instances.ids)
target_group_arn = aws_lb_target_group.core_lb.arn
target_id = each.value
}
Note that you would have to convert the value returned by aws_instances, which is a list, to a set.
From my side I found this solution:
Define a data to collect EC2 IDs from an autoscaling security group:
data "aws_instances" "team_deployment" {
instance_tags = {
Name = local.ec2_name
}
instance_state_names = ["running"]
}
And be sure that instances IDs are the right ones with and evidence (like output)
output "autoscaling_group_ec2_ids" {
value = data.aws_instances.team_deployment.ids
}
Create as many attachments as IDs has found. Use count parameter:
resource "aws_lb_target_group_attachment" "team_deployment" {
count = length(data.aws_instances.team_deployment.ids)
target_group_arn = data.terraform_remote_state.common_resources.outputs.target_group_api.arn
target_id = data.aws_instances.team_deployment.ids[count.index]
port = var.ecr_image_port
depends_on = [data.aws_instances.team_deployment]
}
And problem solved!