I am exploring AWS EC2 autoscaling with target tracking and custom metrics. From the documents I understand that when a particular target is hit, an autoscale event is triggered that either scales in or out the EC2 instance.
I followed the instructions as provided by the terraform docs for aws_autoscaling_policy and is working, but this is scaling in and out just one instance.
Now, for my use-case, i want to scale in and out two instances. Is there a way to do this with target tracking scaling policy?
Any help, much appreciated.
Following is a working policy written in terraform for target tracking with custom metrics.
resource "aws_autoscaling_policy" "target-tracking-autoscale" {
name = "target-traclking-policy"
autoscaling_group_name = "target-tracking-asg"
policy_type = "TargetTrackingScaling"
target_tracking_configuration {
customized_metric_specification {
metric_dimension {
name = "asg"
value = "custom-value"
}
metric_name = "CUSTOM_METRIC"
namespace = "CUSTOM-METRIC/NAMESPACE"
statistic = "Average"
}
target_value = "2"
}
}
Regards.
Update 1
I have tried adding the step_adjustment but this parameter is exclusively for step scaling. Terraform throws this following error:
Error: Error applying plan:
1 error(s) occurred:
* module.pt-wowza.aws_autoscaling_policy.target-tracking-autoscale: 1 error(s) occurred:
* aws_autoscaling_policy.target-tracking-autoscale: step_adjustment is only supported for policy type StepScaling
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
I ran into the same problem and was able to fix it by adding policy_type attribute to the scaling policy resource
policy_type = "TargetTrackingScaling"
policy_type - (Optional) The policy type, either "SimpleScaling", "StepScaling", "TargetTrackingScaling", or "PredictiveScaling". If this value isn't provided, AWS will default to "SimpleScaling."
Related
I use terraform to create an HBase cluster in AWS.
When I use these settings a cluster is provisioned successfully:
resource "aws_emr_cluster" "hbase" {
name = "hbase"
release_label = "emr-6.3.1"
applications = ["HBase"]
termination_protection = false
keep_job_flow_alive_when_no_steps = true
ec2_attributes {
key_name = <removed>
subnet_id = <removed>
instance_profile = aws_iam_instance_profile.emr_profile.arn
}
master_instance_group {
instance_type = "m1.medium"
instance_count = "1"
}
core_instance_group {
instance_type = "m1.medium"
instance_count = 4
ebs_config {
size = "20"
type = "gp2"
volumes_per_instance = 1
}
}
ebs_root_volume_size = 10
As soon as I increase the number of master nodes to three, the cluster creation fails with the error message:
Error: Error waiting for EMR Cluster state to be “WAITING” or “RUNNING”: TERMINATING: BOOTSTRAP_FAILURE: On the master instance (i-), application provisioning timed out
I checked the documentation for aws_emr_cluster, but could not find anything to set a timeout.
I also checked the timeout settings for IAM roles, but the default setting is one hour which would be absolutely sufficient.
https://docs.aws.amazon.com/en_en/IAM/latest/UserGuide/id_roles_use.html
I get the above mentioned error message every time cluster creation takes longer than about 16 minutes (16 minutes and 20 seconds according to the Terraform output).
I have also created an AWS MSK resource in the same project which took longer than 17 minutes. This finished successfully without complaining. So it does not seem like it is a global timeout value.
Any ideas would be much appreciated.
Btw:
terraform version
Terraform v1.1.2
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.60.0
Best,
Denny
The issue has now been resolved. To keep the costs down for this (test) setup I chose instance type "m1.medium", turned out this was the problem.
Using a bigger instance type solved it.
I have 3 existing EBS volumes that I am trying to attach to instances created with Autoscaling groups. Below is Terraform code on how the EBS volumes are defined:
EBS Volumes
resource "aws_ebs_volume" "volumes" {
count = "${(var.enable ? 1 : 0) * var.number_of_zones}"
availability_zone = "${element(var.azs, count.index)}"
size = "${var.volume_size}"
type = "${var.volume_type}"
lifecycle {
ignore_changes = [
"tags",
]
}
tags {
Name = "${var.cluster_name}-${count.index + 1}"
}
}
My plan is to first use the Terraform import utility so the volumes can be managed my Terraform. Without doing this import, Terraform assumes I am trying to create new EBS volumes which I do not want.
Additionally, I discovered this aws_volume_attachment resource to attach these volumes to instances. I'm struggling to determine what value to put as the instance_id in this resource:
Volume Attachment
resource "aws_volume_attachment" "volume_attachment" {
count = length("${aws_ebs_volume.volumes.id}")
device_name = "/dev/sdf"
volume_id = aws_ebs_volume.volumes.*.id
instance_id = "instance_id_from_autoscaling_group"
}
Additionally, the launch configuration block has an ebs_volume_device block, do I need anything else included in this block? Any advice on this matter would be helpful, as I am having some trouble.
ebs_block_device {
device_name = "/dev/sdf"
no_device = true
}
I'm struggling to determine what value to put as the instance_id in this resource
If you create ASG using TF, you don't have access to the instance IDs. The reason is that ASG is treated as one entity, rather then individual instances.
The only way to get the instance ids from ASG created would be through external data resource or lambda function source.
But even if you could theoretically do it, instances in ASG should be treated as fully disposable, interchangeable and identical. You should not need to customize them, as they can be terminated and replaced at any time by AWS's AZ rebalancing, instance failures or scaling activities.
I have created a launch configuration which contains AWS CoreOS AMI as the image. This has been attached into AWS Auto Scaling Group. All the above process has been done via Terraform. But when Auto Scaling group tries to create the instance it fails with following error.
StatusMessage: "In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=ryg425ue2hwnsok9ccfastg4. Launching EC2 instance failed."
It seems like I have to Subscribe to use this CoreOS AMI image, but when I'm creating and instance on AS console, I just select the CoreOS image from market place and continue to other configurations related to instance. But how to achieve this in Terraform? Should I subscribe to AWS CoreOS AMI beforehand or is there a way to bypass this in Terraform?
All the related files and erro trace is given below,
launch-configuration.tf File
resource "aws_launch_configuration" "tomcat-webapps-all" {
name = "tomcat-webapps-all"
image_id = "ami-028e043d0e518a84a"
instance_type = "t2.micro"
key_name = "rnf-sec"
security_groups = ["${aws_security_group.allow-multi-tomcat-webapp-traffic.id}"]
user_data = "${data.ignition_config.webapps.rendered}"
}
auto-scale-group.tf File
resource "aws_autoscaling_group" "tomcat-webapps-all-asg" {
name = "tomcat-webapps-all-asg"
depends_on = ["aws_launch_configuration.tomcat-webapps-all"]
vpc_zone_identifier = ["${aws_default_subnet.default-az1.id}", "${aws_default_subnet.default-az2.id}", "${aws_default_subnet.default-az3.id}"]
max_size = 1
min_size = 0
health_check_grace_period = 300
health_check_type = "EC2"
desired_capacity = 1
force_delete = true
launch_configuration = "${aws_launch_configuration.tomcat-webapps-all.id}"
target_group_arns = ["${aws_lb_target_group.newdasboard-lb-tg.arn}", "${aws_lb_target_group.signup-lb-tg.arn}"]
}
Error Trace
Error: Error applying plan:
1 error(s) occurred:
* aws_autoscaling_group.tomcat-webapps-all-asg: 1 error(s) occurred:
* aws_autoscaling_group.tomcat-webapps-all-asg: "tomcat-webapps-all-asg": Waiting up to 10m0s: Need at least 1 healthy instances in ASG, have 0. Most recent activity: {
ActivityId: "9455ab55-426a-c888-ac95-2d45c78d445a",
AutoScalingGroupName: "tomcat-webapps-all-asg",
Cause: "At 2019-05-20T12:56:29Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.",
Description: "Launching a new EC2 instance. Status Reason: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=ryg425ue2hwnsok9ccfastg4. Launching EC2 instance failed.",
Details: "{\"Subnet ID\":\"subnet-c650458f\",\"Availability Zone\":\"ap-southeast-1a\"}",
EndTime: 2019-05-20 12:56:30 +0000 UTC,
Progress: 100,
StartTime: 2019-05-20 12:56:30.642 +0000 UTC,
StatusCode: "Failed",
StatusMessage: "In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=ryg425ue2hwnsok9ccfastg4. Launching EC2 instance failed."
}
If you log into the console and accept the ULA terms once this error will go away when you apply it via terraform.
If I was you I'd log in, go through the whole process to launch an instance with this AMI, terminate it, then apply the terraform.
If somebody is also having the same issue, I was able to solve it by login into my EC2 console with root user and subscribing to AWS CoreOS Product Page on AWS Marketplace.
After that everything worked as expected. The error returned with a web URL to CoreOS product page on AWS Marketplace. Its just a matter of clicking Continue to Subscribe Button.
If above steps didn't work refer this answer - https://stackoverflow.com/a/56222898/4334340
I am trying to create ec2 instance through auto_scaling_group on terraform
I have something like:
resource "aws_ecs_cluster" "my_cluster" {
name = "my-cluster"
}
resource "aws_autoscaling_group" "my_instances" {
name = "my-instances"
min_size = 1
max_size = 2
availability_zones = ["us-east-1a"]
launch_configuration = "${aws_launch_configuration.my_ecs_instance.id}"
}
resource "aws_launch_configuration" "my_ecs_instance" {
name_prefix = "my-ecs-instance"
instance_type = "t2.micro"
image_id = "ami-19e8cc0e"
}
Terraform plan -var-file=mykey.tfvars
works fine but
Terraform apply -var-file=mykey.tfvars
will stock in creating the instance like
aws_autoscaling_group.my_instances: Still creating... (9m20s elapsed)
aws_autoscaling_group.my_instances: Still creating... (9m30s elapsed)
aws_autoscaling_group.my_instances: Still creating... (9m40s elapsed)
eventually time out and saying
aws_autoscaling_group.my_instances: "my-instances"
Waiting up to 10m0s: Need at least 1 healthy instances in ASG, have 0. Most recent activity:
..more..
StatusMessage: "No default VPC for this user. Launching EC2 instance failed."
I think I need to specify vpc id but I don't find auto_scaling_group has vpc_id attribute.
I am not sure how to fix this, can someone help me about it? Thanks a lot!
This wait is because the autoscalling group is waiting for at least one ec2 instance to up and running as defined in the auto scaling group but there is none. This resulted in the error which mentioned the root cause "No default VPC for this user". So basically, there is no ec2 up and runing because there is no VPC, subnet and/or VPC identifier associated with the autoscaling group.
To resolve:
First if you haven't done this, you will need to create a VPC with the vpc resource "aws_vpc"
Next create a subnet with the subnet resources "aws_subnet"
Next associate the VPC identifier "vpc_zone_identifier" with the auto scaling group in "aws_autoscaling_group" resource area
The identifier should look like below where "aws_subnet.main-public-1" is the subnet ID created in step 2
vpc_zone_identifier = ["${aws_subnet.main-public-1.id}"
I hope that helps
The error message is: StatusMessage: "No default VPC for this user. Launching EC2 instance failed."
You need to create VPC with subnets and provide the subnet ids when creating autoscaling group.
Think about to add vpc_zone_identifier
vpc_zone_identifier (Optional) A list of subnet IDs to launch resources in.
https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html#vpc_zone_identifier
I want to create an ASG with only 1 instance initially.
I want all the instances of this ASG to be behind an ELB.
So I set load_balancers = ["${aws_elb.Production-Web-ELB.name}"] in the resource "aws_autoscaling_group" "ProductionWeb-ScalingGroup" .
Now, when I write the code for the resource "aws_elb" "Production-Web-ELB", and I set instances = ["${aws_autoscaling_group.ProductionWeb-ScalingGroup.*.id}"], I get the error...
Error configuring: 1 error(s) occurred:
* Cycle: aws_autoscaling_group.ProductionWeb-ScalingGroup, aws_elb.Production-Web-ELB
I understand that this error means that the one resource references the other in a circle. To check it I comment out the load_balancers = ["${aws_elb.Production-Web-ELB.name}"] part and terraform plan without any error.
So my question is: Am I unable using Terraform to create an ASG with an attached ELB and every EC2 that will spawn inside it will be automatically behind the ELB ?
Is there something from the documentation that I missed?
Is there a workaround?
You don't need to explicitly define the instances that will be associated with the ELB in terraform's ELB definition. By using the load_balancers argument, you're associating the ELB with the AutoScaling group, and AutoScaling will know to attach any instances that are created to that ELB when the AutoScaling group launches that instance.
Terraform isn't directly managing the state of the instances in this case -- AWS AutoScaling is, so their state likewise don't need to be defined in terraform beyond defining a launch configuration and associating it to the AutoScaling group.
To tell terraform to launch the AutoScaling group with a single instance, set your min_size argument to 1 and let your scaling policies handle the desired capacity from there. You could alternatively set desired_capacity to 1, but be wary of managing that state in terraform because it will set the desired_capacity to 1 every time you apply your plan.