I am creating ecs taskDefinition using terraform.
resource "aws_ecs_task_definition" "tktest_terraform-td" {
family = "nodejs-webapp"
container_definitions = "${templatefile("${path.module}/taskdefinition/service-td.tpl", { webapp_docker_image = "${var.webapp_docker_image_name}:${var.webapp_docker_image_tag}"})}"
}
When ever there is a changes to the taskdefinition a new revision is created but the problem is the older revision is gets deleted.
Is it possible to create a new revision but the same time preserve the older revision?
This is long-lasting issue with task definition documented and discussed on GitHub Issues.
So far, the issue is open, and the workaround reported is to manually remove current version of the task from the TF state. In your case it would be:
# we can still get the task definition diff at this point, which we care about
terraform plan
# remove from state so that task definition is not destroyed, and we're able to rollback in the future if needed
terraform state rm aws_ecs_task_definition.tktest_terraform-td
# diff will show a brand new task definition created, but that's ok because we got the diff in step 1
terraform apply
I have been playing around with using the terraform aws provider to create managed services such as Amazon MQ. I found this worked nicely overall but when testing changing the minor version (5.15.9 -> 5.15.10) this is a destructive process which would result in data loss if this was being performed on a live service.
I have read the documentation for the provisioner and did not any further config that would allow this to happen without destroying and recreating the broker(s). The only option I could see would be to manually update via AWS directly or allow minor updates to happen automatically.
Any feedback on if this can be done via Terraform? Otherwise I would imagine the best process might be some automation or manual pipeline using the AWS CLI to do upgrades but then the terraform state will be stale I would presume.
Edit:
See terraform for creating the broker below, I have removed content related to secret manager, security groups, subnets etc as it seemed to add bulk and not be related to the issue directly.
resource "aws_mq_broker" "broker" {
broker_name = "${var.env}_${var.team}_broker"
engine_type = "ActiveMQ"
engine_version = "5.15.9"
deployment_mode = "${lookup(var.team_env_config, "${var.env}_${var.team}_deployment_mode")}"
host_instance_type = "${lookup(var.team_env_config, "${var.env}_${var.team}_instance_type")}"
auto_minor_version_upgrade = false
}
When testing the upgrade I just changed 5.15.9 -> 5.15.10
The output is as follows:
deployment_mode: "SINGLE_INSTANCE" => "SINGLE_INSTANCE"
engine_type: "ActiveMQ" => "ActiveMQ"
engine_version: "5.15.9" => "5.15.10" (forces new resource)
That is just a snipped but as you can see the engine version change forces a new resource.
When ever I am running a terraform plan using the following:
resource "google_sql_user" "users" {
name = "me"
instance = "${google_sql_database_instance.master.name}"
host = "me.com"
password = "changeme"
}
This only happens when I am running this a Postgres instance on Google Cloud SQL.
Terraform always outputs that the plan will create a user, even though it's already created. I'm using terraform version 0.11.1. Is there something that i'm missing? I tried setting the id value, however there it still recretes itself.
It turns out that the terraform state of the user was not being added, the cause seemed to be that when using postgres, the value host = "me.com" is not required, but causes problems if it's left there.
Once that was removed, then the terraform state will be correct.
I am new to Terraform. I am using Terraform to write AWS scripts. I am getting an error while performing Terraform Destroy. Terraform script is
resource "aws_rds_cluster" "aurora-cluster-ci" {
cluster_identifier = "aurora-cluster-ci"
engine = "aurora-mysql"
availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"]
database_name = "${var.rds_dbname}"
master_username = "${var.rds_username}"
master_password = "${var.rds_password}"
backup_retention_period = 5
engine_version = "5.7.16"
preferred_backup_window = "07:00-09:00"
apply_immediately = true
final_snapshot_identifier = "ci-aurora-cluster-backup"
skip_final_snapshot = true
}
Terraform Destroy throws an error "aws_rds_cluster.aurora-cluster-ci: RDS Cluster
FinalSnapshotIdentifier is required when a final snapshot is required"
I have "final_snapshot_identifier" key in my script.
Solution:
I Encountered the same problem while trying to perform a destroy on an RDS instance (not under AWS Aurora) but the principles are the same.
Below are a few steps I took in order to solve this issue:
Change skip_final_snapshot to true and remove final_snapshot_identifier if exists (see comments #1 and #2 below) .
Remove backup_window (Under AWS Aurora its probably called preferred_backup_window).
Change backup_retention_period to 0.
Make sure that apply_immediately is set to true (see comment #3 below).
Run terraform apply and check the changes to affect (see a tip as comment #4 below).
Now you can run terraform destroy and no errors should appear (in my case I add deletion_protection set to true and add to remove it).
Comment #1 - Understanding the the purpose of the relevant fields
From Terraform docs:
skip_final_snapshot - (Optional) Determines whether a final DB snapshot is created before the DB instance is deleted. If true is specified, no DBSnapshot is created. If false is specified, a DB snapshot is created before the DB instance is deleted, using the value from final_snapshot_identifier. Default is false.
final_snapshot_identifier - (Optional) The name of your final DB snapshot when this DB instance is deleted. Must be provided if skip_final_snapshot is set to false.
In the code specified in the question skip_final_snapshot was true and final_snapshot_identifier was still specified.
(*) Don't be confused with the snapshot_identifier field.
Comment #2 - What is causing this error?
For those who want to understand a little bit what is happening here, in the mentioned open issue there is a nice thread where a contributor named #caiges gave a nice explanation there:
For starters,
skip_final_snapshot defaults to False which should also require
final_snapshot_identifier to be set but it's not so what happens is
the create/update is applied, state updated where skip_final_snapshot
is False but final_snapshot_identifier is null.
This causes the
destroy operation to fail it's verification stage.
This can be fixed but I don't really have a great story for those who
already have prexisting state. One possibility would be that a delete
operation ignores skip_final_shopshot if the identifier is null.
Another might be to default final_snapshot_identifier to something
random if skip_final_snapshot is set to or defaulted to False. I think
for data safety reasons, ignoring skip_final_snapshot if
final_snapshot_identifier is null is a bad idea and it'd be better to
just randomize an identifier.
Comment #3 - Making sure our changes take immediate effect:
A note about apply_immediately from Terraform's docs:
Note: using apply_immediately can result in a brief downtime as the
server reboots. See the AWS Docs on RDS Maintenance for more
information.
Comment #4 (Bonus) - Saving ourselves some time:
When you run terraform plan make sure that the ~ (update in-place sign) appears in the relevant fields under Terraform's execution plan - In the example below you can see that 2 changes will be applied:
~ resource "aws_db_instance" "postgresql" {
address = ...
allocated_storage = 100
allow_major_version_upgrade = false
.
.
~ apply_immediately = false -> true
.
.
~ backup_retention_period = 7 -> 0
.
.
tags = ...
username = ...
vpc_security_group_ids = ...
}
This might sound trivial, but in cases like this error, it can save a lot of debugging time when you try to understand why certain updates haven't took place.
This is a known bug that is still open as of the current version of the Terraform provider for AWS:
https://github.com/terraform-providers/terraform-provider-aws/issues/2588
In a nutshell, it's ignoring the skip_final_snapshot parameter.
In my case I had to manually edit the .tfstate file and set "skip_final_snapshot" to true. Then it worked.
To delete RDS DB from terraform destroy:-
first add skip_final_snapshot = "true" to your aws_provider
do terraform-apply
Then you are able to destroy it.
terraform destroy
If you're a Pulumi user seeing this error as Pulumi uses the Terraform provider:
pulumi stack export > export.json
Then change all instances of skipFinalSnapshot to true.
And import the changed file:
pulumi stack import --file export.json
Ran into the same issue, being a starter with RDS terraform resource you may miss skip_final_snapshot=true/false flag. By default this will be in false state and when you go for a terraform destroy it expects a snapshot name of the db causing the error.
*Name can be given with final_snapshot_identifier flag, if you want to create a final snap.
But now when you have created the RDS instance without knowing which obviously you wont.
just remove the state of that RDS instance
in my case:
1.) terraform state rm module.rds.aws_rds_instance_default
2.) Manually delete the RDS instance from the AWS console.
3.) reapply with terraform apply with skip_final_snapshot=true in aws_rds_instance resource.
or in case you want to create instance snap when you destroy it.
set skip_final_snapshot=false, final_snapshot_identifier=name-of-snapshot.
Hope this helps !!Thanks
I had the same issue but now I can destroy it after I changed the skipFinalSnapshot to true and backup_retention_period to 0 then I did terraform apply. Once these changes are applied and run the terraform destroy command. It will work.
If you want to update the db which needs the replacement of resource and you are getting an error of "RDS Cluster FinalSnapshotIdentifier is required when a final snapshot is required" then first comment out the parameters due to which replacement is required and add below parameter:
skip_final_snapshot = true
Then do "terraform apply".
Post that apply the updates which you want and run "terraform apply"
This solved my issue.
I was not able to delete an rds instance that I created through terraform scripts. Then I realized - It is not only enough to keep the skip_final_snapshot to true but also to do a terraform apply so that the changed value is taken into consideration. After that a terraform destroy does a proper deletion of the resources without the error 'Error: DB Instance FinalSnapshotIdentifier is required when a final snapshot is required'
I ran into this problem because I was trying to make a change that forced replacement of the instance, while also trying to set set_final_snapshot to true.
Make sure that you run apply set_final_snapshot = true before applying the subsequent replacement-forcing change (two applys in total).
I am trying to create a DMS task to migrate data from RDS Postgres instance to S3 bucket. The full load works all fine, but the continuous replication is failing. Its giving this error:
"logical decoding requires wal_level >= logical"
When I checked the system settings from pg_settings, its showing that the setting "wal_level" has value "replica". So I tried to change the setting wal_level, but I am not able to find this setting in the Parameter Group in RDS. My RDS instance is using 9.6 version of parameters.
When I tried "ALTER SYSTEM SET wal_level TO 'logical'", it fails saying that "must be superuser to execute ALTER SYSTEM command", even though the user is under rds_superuser role.
Please suggest.
The Parameter name in Parameter Group is "rds.logical_replication" which needs to be changed to 1. The default value was 0.
This property changed "wal_level" value to "Logical".