Terraform Bigquery create tables replace table instead of edit - google-cloud-platform

Question

I added a json file that contains all the tables i want to create :
tables.json
"tables": {
"table1": {
"dataset_id": "dataset1",
"table_id": "table1",
"schema_path": "folder/table1.json"
},
"table2": {
"dataset_id": "dataset2",
"table_id": "table2",
"schema_path": "folder/table2.json"
}
}
Then with a foreach on a Terraform resource, i want to create these tables dynamically :
local.tf file
locals {
tables = jsondecode(file("${path.module}/resource/tables.json"))["tables"]
}
variables.tf file
variable "project_id" {
description = "Project ID, used to enforce providing a project id."
type = string
}
variable "time_partitioning" {
description = "Configures time-based partitioning for this table."
type = map(string)
default = {
type = "DAY"
field = "my_field"
}
}
main.tf file
resource "google_bigquery_table" "tables" {
for_each = local.tables
project = var.project_id
dataset_id = each.value["dataset_id"]
table_id = each.value["table_id"]
dynamic "time_partitioning" {
for_each = [
var.time_partitioning
]
content {
type = try(each.value["partition_type"], time_partitioning.value["type"])
field = try(each.value["partition_field"], time_partitioning.value["field"])
expiration_ms = try(time_partitioning.value["expiration_ms"], null)
require_partition_filter = try(time_partitioning.value["require_partition_filter"], null)
}
}
schema = file("${path.module}/resource/schema/${each.value["schema_path"]}")
}
The schema files contains classic bigquery schema, for example :
[
{
"name": "field",
"type": "STRING",
"mode": "NULLABLE",
"description": "My field"
}
]
The creation of tables works well, but when i add a new nullable field on a schema, Terraform proposes to "replace table" (destroy and recreate) instead of "update table".
The normal behaviour in this case for native Bigquery and Terraform is to update the table.
When a do the same test with the same Terraform resource but without forEach, Terraform has the expected behaviour and proposes to update the table.
An example of the Terraform log with "forEach" :
# google_bigquery_table.tables["table1"] must be replaced
-/+ resource "google_bigquery_table" "tables" {
~ creation_time = 1616764894477 -> (known after apply)
dataset_id = "dataset1"
deletion_protection = true
~ etag = "G9qwId8jgQS8nN4N61zqcA==" -> (known after apply)
~ expiration_time = 0 -> (known after apply)
~ id = "projects/my-project/datasets/dataset1/tables/table1" -> (known after apply)
- labels = {} -> null
~ last_modified_time = 1617075251337 -> (known after apply)
~ location = "EU" -> (known after apply)
~ num_bytes = 0 -> (known after apply)
~ num_long_term_bytes = 0 -> (known after apply)
~ num_rows = 0 -> (known after apply)
project = "project"
~ schema = jsonencode(
~ [ # forces replacement
{
description = "Field"
mode = "NULLABLE"
name = "field"
type = "STRING"
}
.....
+ {
+ description = "Field"
+ mode = "NULLABLE"
+ name = "newField"
+ type = "STRING"
}
Terraform displays and detects correctly the new column to add for a table, but indicates a replace instead of an edition.
I repeat that, the exact same test with a the same Terraform resource without forEach and on a single Bigquery table, works well (same schema, same change). I create the table and when a add a new nullable column, Terraform proposes an edition (the expected behaviour).
I checked on Terraform doc and web, i didn't saw some examples to manage a list of table with Terraform.
Is it not possible to create and update tables with configured tables and foreach ?
Thanks for your help.

Answer 1

This sounded like a provider bug. I found this issue in the terraform-provider-google repository that seems related to your issue. The fix was merged just 13 hours ago (at the time of writing). So, maybe you can wait for the next release (v3.63.0) and see if it fixes your issue.
Just FYI: You might want to verify that the fix commit was actually included in the next release. It happened to me before that something that was merged in master before a released was not actually released.

Answer 2

Thanks so much #Alessandro, the problem was indeed due to the Terraform provide Google version.
I used the v3.62.0 version of Google provider, and you target me to the good direction.
I saw this link too : https://github.com/hashicorp/terraform-provider-google/issues/8503
There is a very useful comment by "tpolekhin" (thanks to him) :
Hopefully im not beating a dead horse commenting on the closed issue, but I did some testing with various versions on the provider, and it behaves VERY differently each time.
So, our terraform code change was pretty simple: add 2 new columns to existing BigQuery table SCHEDULE
Nothing changes between runs - only provider version
v3.52.0
Plan: 0 to add, 19 to change, 0 to destroy.
Mostly adds + mode = "NULLABLE" to fields in bunch of tables, and adds 2 new fields in SCHEDULE table
v3.53.0
Plan: 0 to add, 2 to change, 0 to destroy.
Adds 2 new fields to SCHEDULE table, and moves one field in another table in a different place (sorting?)
v3.54.0
Plan: 1 to add, 1 to change, 1 to destroy.
Adds 2 new fields to SCHEDULE table, and moves one field in another table in a different place (sorting?) but now with table re-creation for some reason
v3.55.0
Plan: 0 to add, 2 to change, 0 to destroy.
Adds 2 new fields to SCHEDULE table, and moves one field in another table in a different place (sorting?)
behaves exactly like v3.53.0
v3.56.0
Plan: 1 to add, 0 to change, 1 to destroy.
In this comment, we can see that some versions have the problem.
For example this works with v3.55.0 but not with v3.56.0
I temporary downgrade the version to v3.55.0 and when the next release will solve this issue, i will upgrade it.
provider.tf :
provider "google" {
version = "= 3.55.0"
}