I'm quite new to Terraform, and struggling with something.
I'm playing around with Redshift for a personal project, and I want to update the inbound security rules for the default security group which is applied to Redshift when it's created.
If I were doing it in AWS Console, I'd be adding a new inbound rule with Type being All Traffic and Source being Anywhere -IPv4 which adds 0.0.0.0/0.
Below in main.tf I've tried to create a new security group and apply that to Redshift, but I get a VPC-by-Default customers cannot use cluster security groups error.
What is it I'm doing wrong?
resource "aws_redshift_cluster" "redshift" {
cluster_identifier = "redshift-cluster-pipeline"
skip_final_snapshot = true terraform destroy
master_username = "awsuser"
master_password = var.db_password
node_type = "dc2.large"
cluster_type = "single-node"
publicly_accessible = "true"
iam_roles = [aws_iam_role.redshift_role.arn]
cluster_security_groups = [aws_redshift_security_group.redshift-sg.name]
}
resource "aws_redshift_security_group" "redshift-sg" {
name = "redshift-sg"
ingress {
cidr = "0.0.0.0/0"
}
The documentation for the Terraform resource aws_redshift_security_group states:
Creates a new Amazon Redshift security group. You use security groups
to control access to non-VPC clusters
The error message you are receiving is clearly staging that you are using the wrong type of security group, and you need to use a VPC security group instead. Once you create the appropriate VPC security group, you would set it in the aws_redshift_cluster resource via the vpc_security_group_ids property.
Related
I'm following terraform documentation to create Aurora serverless v2 by terraform. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/rds_cluster#rds-serverless-v2-cluster
In terraform documentation, they have not mentioned how to create Aurora serverless v2 with multi AZ (read replica in other region for failover). Although, by aws console, I can create multi AZ Aurora serverless v2.
Any help is appreciated to create Aurora serverless v2 with multi AZ by terraform?
You can't set it up becasuse by default your data is stored across multiple AZs. From docs:
The storage for each Aurora DB cluster consists of six copies of all your data, spread across three AZs. This built-in data replication applies regardless of whether your DB cluster includes any readers in addition to the writer. That way, your data is safe, even from issues that affect the compute capacity of the cluster.
I found a solution. I am using a custom module created from https://github.com/terraform-aws-modules/terraform-aws-rds-aurora and also I saw one more link https://github.com/hashicorp/terraform-provider-aws/issues/24502. In this link, check the answer from msbainuk. You have to define additional instance, which will automatically create it as a Multi AZ. PFB the Code snippet from his answer.
resource "aws_rds_cluster_instance" "cluster_instances" {
cluster_identifier = aws_rds_cluster.this.id
instance_class = "db.serverless"
engine = aws_rds_cluster.this.engine
engine_version = aws_rds_cluster.this.engine_version
}
Hope this helps.
You should create multiple aws_rds_cluster_instances. Here's an example from the docs:
resource "aws_rds_cluster_instance" "cluster_instances" {
count = 2
identifier = "aurora-cluster-demo-${count.index}"
cluster_identifier = aws_rds_cluster.default.id
instance_class = "db.r4.large"
engine = aws_rds_cluster.default.engine
engine_version = aws_rds_cluster.default.engine_version
}
resource "aws_rds_cluster" "default" {
cluster_identifier = "aurora-cluster-demo"
availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
database_name = "mydb"
master_username = "foo"
master_password = "barbut8chars"
}
I believe the AWS console UI is masking over that. It's going to create 2x instances if you select "multi-az".
ok, so I am trying to attach an EBS volume which I have created using Terraform to an ASG's instance using userdata, but now issue is both are in different AZ's, due to which, it failing to attach. Below is the steps I am trying and failing:
resource "aws_ebs_volume" "this" {
for_each = var.ebs_block_device
size = lookup(each.value,"volume_size", null)
type = lookup(each.value,"volume_type", null)
iops = lookup(each.value, "iops", null)
encrypted = lookup(each.value, "volume_encrypt", null)
kms_key_id = lookup(each.value, "kms_key_id", null)
availability_zone = join(",",random_shuffle.az.result)
}
In above resource, I am using random provider to get one AZ from list of AZs, and same list is provided to ASG resource below:
resource "aws_autoscaling_group" "this" {
desired_capacity = var.desired_capacity
launch_configuration = aws_launch_configuration.this.id
max_size = var.max_size
min_size = var.min_size
name = var.name
vpc_zone_identifier = var.subnet_ids // <------ HERE
health_check_grace_period = var.health_check_grace_period
load_balancers = var.load_balancer_names
target_group_arns = var.target_group_arns
tag {
key = "Name"
value = var.name
propagate_at_launch = true
}
}
And here is userdata which I am using:
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
instanceId = curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
aws ec2 attach-volume --volume-id ${ebs_volume_id} --instance-id $instanceId --device /dev/nvme1n1
Above will attach the newly created volume, as I am passing output ${ebs_volume_id} of above resource.
But, its failing because instance and volume are in different AZs.
Can anyone help me on this as a better solution than hardcoding AZ on both ASG and Volume?
I'd have to understand more about what you're trying to do to solve this with just the aws provider and terraform. And honestly, most ideas are going to be a bit complex.
You could have an ASG per AZ. Otherwise, the ASG is going to select some AZ at each launch. And you'll have more instances in an AZ than you have volumes and volumes in other AZs with no instances to attach to.
So you could create a number of volumes per az and an ASG per AZ. Then the userdata should list all the volumes in the AZ that are not attached to an instance. Then pick the id of the first volume that is unattached. Then attach it. If all are attached, you should trigger your alerting because you have more instances than you have volumes.
Any attempt to do this with a single ASG is really an attempt at writing your own ASG but doing it in a way that fights with your actual ASG.
But there is a company who offers managing this as a service. They also help you manage them as spot instances to save cost: https://spot.io/
The elastigroup resource is an ASG managed by them. So you won't have an aws asg anymore. But they have some interesting stateful configurations.
We support instance persistence via the following configurations. all values are boolean. For more information on instance persistence please see: Stateful configuration
persist_root_device - (Optional) Boolean, should the instance maintain its root device volumes.
persist_block_devices - (Optional) Boolean, should the instance maintain its Data volumes.
persist_private_ip - (Optional) Boolean, should the instance maintain its private IP.
block_devices_mode - (Optional) String, determine the way we attach the data volumes to the data devices, possible values: "reattach" and "onLaunch" (default is onLaunch).
private_ips - (Optional) List of Private IPs to associate to the group instances.(e.g. "172.1.1.0"). Please note: This setting will only apply if persistence.persist_private_ip is set to true
stateful_deallocation {
should_delete_images = false
should_delete_network_interfaces = false
should_delete_volumes = false
should_delete_snapshots = false
}
This allows you to have an autoscaler that preserves volumes and handles the complexities for you.
I am trying to spin up an ECS cluster with Terraform, but can not make EC2 instances register as container instances in the cluster.
I first tried with the verified module from Terraform, but this seems out dated (ecs-instance-profile has wrong path).
Then I tried with another module from anrim, but still no container instances. Here is the script I used:
provider "aws" {
region = "us-east-1"
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "2.21.0"
name = "ecs-alb-single-svc"
cidr = "10.10.10.0/24"
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
private_subnets = ["10.10.10.0/27", "10.10.10.32/27", "10.10.10.64/27"]
public_subnets = ["10.10.10.96/27", "10.10.10.128/27", "10.10.10.160/27"]
tags = {
Owner = "user"
Environment = "me"
}
}
module "ecs_cluster" {
source = "../../modules/cluster"
name = "ecs-alb-single-svc"
vpc_id = module.vpc.vpc_id
vpc_subnets = module.vpc.private_subnets
tags = {
Owner = "user"
Environment = "me"
}
}
I then created a new ecs cluster (from the aws console) on the same VPC and carefully compared the differences in resources. I managed to find some small differences, fixed them and tried again. But still no container instances!
A fork of the module is available here.
Can you see instances being created in the autoscaling group? If so, I'd suggest SSHing to one of them (either directly or using a bastion host, eg. see this module) and checking ECS agent logs. In my experience those problems are usually related to IAM policies, and that's pretty visible in logs but YMMV.
I'm brand new to Terraform so I'm sure i'm missing something, but the answers i'm finding don't seem to be asking the same question I have.
I have an AWS VPC/Security Group that we need our EC2 instances to be created under and this VPC/SG is already created. To create an EC2 instance, Terraform requires that if I don't have a default VPC, I must import my own. But once I import and apply my plan, when I wish to destroy it, its trying to destroy my VPC as well. How do I encapsulate my resources so when I run "terraform apply", I can create an EC2 instance with my imported VPC, but when I run "terraform destroy" I only destroy my EC2 instance?
In case anyone wants to mention, I understand that:
lifecycle = {
prevent_destroy = true
}
is not what I'm looking for.
Here is my current practice code.
resource "aws_vpc" "my_vpc" {
cidr_block = "xx.xx.xx.xx/24"
}
provider "aws" {
region = "us-west-2"
}
data "aws_ami" "ubuntu" {
most_recent = true
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*"]
}
owners = ["099720109477"] # Canonical
}
resource "aws_instance" "web" {
ami = "${data.aws_ami.ubuntu.id}"
instance_type = "t3.nano"
vpc_security_group_ids = ["sg-0e27d851dxxxxxxxxxx"]
subnet_id = "subnet-0755c2exxxxxxxx"
tags = {
Name = "HelloWorld"
}
}
Terraform should not require you to deploy or import a VPC in order to deploy an EC2 instance into it. You should be able to reference the VPC, subnets and security groups by id so TF is aware of your existing network infrastructure just like you've already done for SGs and subnets. All you should need to deploy the EC2 instance "aws_instance" is give it an existing subnet id in the existing VPC like you already did. Why do you say deploying or importing a VPC is required by Terraform? What error or issue do you have deploying without the VPC and just using the existing one?
You can protect the VPC through AWS if you really wanted to, but I don't think you really want to import the VPC into your Terraform state and let Terraform manage it here. Sounds like you want the VPC to service other resources, maybe applications manually deployed or through other TF stacks, and the VPC to live independent of anyone application deployment.
To answer the original question, you can use a data source and match your VPC by id or tag name :
data "aws_vpc" "main" {
tags = {
Name = "main_vpc"
}
}
Or
data "aws_vpc" "main" {
id = "vpc-nnnnnnnn"
}
Then refer to it with : data.aws_vpc.main
Also, if you already included your VPC but would like not to destroy it while remove it from your state, you can manage to do it with the terraform state command : https://www.terraform.io/docs/commands/state/index.html
Trying to learn to use Terraform (v 0.3.7) with Amazon Web Services.
When I create a VPC using Terraform via the following:
resource "aws_vpc" "test-vpc" {
cidr_block = "${var.vpc_cidr}"
enable_dns_hostnames = true
tags {
Name = "test-vpc"
}
}
The VPC will have a main routing table and a "default" security group automatically created (I assume by AWS, rather than Terraform); these can be identified by the attributes on the created VPC: main_route_table_id and default_security_group_id.
While following this tutorial it talks about creating your own default security group and routing table - it makes no mention of the default ones that will get created (even if you create your own routing table, the "main" one created by default will just remain sitting there, associated with no subnets or anything).
Shouldn't we be using the default resource that created with a VPC? Especially the routing table, will there be any effects because of not using the "main" routing table?
And if I should be using the default resources, how do I do that with Terraform?
I couldn't see anything in the Terraform documentation about these default resources, and if I try to override them (for example by telling Terraform to create a security group with name default, I get errors).
AWS creates these default routing tables and sec groups. If you don't use them ( I know we don't) they are fine to get deleted.
Terraform throws errors if you require it to create default sec group as probably the group is already there or maybe this sec group name is reserved.
You can create one new resource "aws_security_group" ( https://terraform.io/docs/providers/aws/r/security_group.html )and have a dependency listed on the resource with
depends_on = ["aws_instance.instance-name-from-resource"]
for the instance thus sec group will be created first and then assign sec groups to the instance with "security_groups"