Terraform - Cyclic dependency issue on GCP - google-cloud-platform

I am provisioning multiple resources on GCP including a Cloud SQL (Postgres) DB and one VM instance. I am struggling with a cyclic dependency on Terraform during terraform apply as:
Cloud SQL (Postgres) needs the IP of the VM for IP whitelisting
The VM uses a start-up script that requires the Public IP of the Postgres DB
Hence, the cyclic dependency... Do you have any suggestion to tackle this in Terraform?
File that creates the GCP VM (includes a startup script that requires the IP of the Postgres DB)
data "template_file" "startup_script_airbyte" {
template = file("${path.module}/sh_scripts/airbyte.sh")
vars = {
db_public_ip = "${google_sql_database_instance.postgres.public_ip_address}"
db_name_prefix = "${var.db_name}"
db_user = "${var.db_user}"
db_password = "${var.db_password}"
}
}
resource "google_compute_instance" "airbyte_instance" {
name = "${google_project.data_project.project_id}-airbyte"
machine_type = local.airbyte_machine_type
project = google_project.data_project.project_id
metadata_startup_script = data.template_file.startup_script_airbyte.rendered #file("./sh_scripts/airbyte.sh")
allow_stopping_for_update = true
depends_on = [
google_project_service.data_project_services,
]
boot_disk {
initialize_params {
image = "ubuntu-2004-focal-v20210415"
size = 50
type = "pd-balanced"
}
}
network_interface {
network = "default"
access_config {
network_tier = "PREMIUM"
}
}
service_account {
email = google_service_account.airbyte_sa.email
scopes = ["cloud-platform"]
}
}
Script that creates the Postgres DB (requires IP of the VM above)
resource "google_sql_database_instance" "postgres" {
name = "postgres-instance-${random_id.db_name_suffix.hex}"
project = google_project.data_project.project_id
database_version = "POSTGRES_13"
settings{
tier = "db-f1-micro"
backup_configuration {
enabled = true
start_time = "02:00"
}
database_flags {
name = "cloudsql.iam_authentication"
value = "on"
}
database_flags {
name = "max_connections"
value = 30000
}
#Whitelisting the IPs of the GCE VMs in Postgres
ip_configuration {
ipv4_enabled = "true"
authorized_networks {
name = "${google_compute_instance.airbyte_instance.name}"
value = "${google_compute_instance.airbyte_instance.network_interface.0.access_config.0.nat_ip}"
}
}
}
}

One way to overcome this would be to get static public IP, using google_compute_address. You do this before you create your instance, and then attach it to the instance.
This way the IP can be whitelisted in Cloud SQL, before the instance is created.

The correct solution is to install the Cloud SQL Auth Proxy in the VM. Then you do not need to whitelist IP addresses. This will remove the cyclic dependency.

Related

Access denied issue with GCP SQL created using Terraform

I created following resources using the terraform
MySql Instance
DB
User
To create these resources I am using GoogleCloudPlatform/sql-db/google//modules/mysql v13.0.1
The Terraform template is as follow:
module "backend-db" {
source = "GoogleCloudPlatform/sql-db/google//modules/mysql"
version = "13.0.1"
name = "${var.env}-database"
random_instance_name = true
database_version = "MYSQL_5_7"
project_id = module.dev-project.project_id
region = "australia-southeast1"
zone = "australia-southeast1-a"
tier = "db-n1-standard-1"
additional_databases = var.additional_databases
additional_users = var.additional_users
deletion_protection = false
ip_configuration = {
ipv4_enabled = true
private_network = null
require_ssl = true
allocated_ip_range = null
authorized_networks = local.authorized_networks
}
database_flags = [
{
name = "log_bin_trust_function_creators"
value = "on"
},
]
}
The Instance has public IP and in authorized network, I added my IP and I am trying to access this via MySql Workbench.
The Error is get is Access denied for user '<USER>'#'<MY_IP>' (using password: YES)
Note: All works well when I create all the resource from the console (UI)
Please help.

I have a Terraform that creates a AWS RDS, aurora mysql, is there a way to create a table on the DB

I have a Terraform script that creates a AWS RDS, aurora mysql cluster
module "cluster" {
source = "terraform-aws-modules/rds-aurora/aws"
name = var.cluster_name
master_username = var.master_username
master_password = var.master_password
create_random_password = false
database_name = var.database_name
engine = var.engine
engine_version = var.engine_version
instance_class = var.instance_class_r5
instances = {
one = {}
2 = {
instance_class = var.instance_class_r5_2
}
}
vpc_id = var.vpc_id
subnets = ["subnet-XXXX", "subnet-XXXX", "subnet-XXXX"]
allowed_security_groups = ["sg-XXXXXXXXXXXXXX"]
allowed_cidr_blocks = ["10.20.0.0/20", "144.121.18.66/32"]
storage_encrypted = true
apply_immediately = true
monitoring_interval = 10
db_parameter_group_name = aws_db_parameter_group.credential.id
db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.credential.id
publicly_accessible = true
}
resource "aws_db_parameter_group" "credential" {
name = "${var.cluster_name}-aurora-db-57-parameter-group"
family = "aurora-mysql5.7"
description = "${var.cluster_name}-aurora-db-57-parameter-group"
tags = var.tags_required
}
resource "aws_rds_cluster_parameter_group" "credential" {
name = "${var.cluster_name}-aurora-57-cluster-parameter-group"
family = "aurora-mysql5.7"
description = "${var.cluster_name}-aurora-57-cluster-parameter-group"
tags = var.tags_required
}
This creates a database
I am using springboot, and usually with a databse the entity will create the table
#Entity
#Table(name="credential")
public class CredentialEntity {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
Long credentialId;
In my yml file I have set
spring:
hibernate:
ddl-auto: update
But it does not create the table. So is there a way to create the table as part of the terraform script.
I wouldn't recommend doing this, but if you want Terraform to deploy database structures you can try with:
resource "null_resource" "db_setup" {
depends_on = [module.db, aws_security_group.rds_main, aws_default_security_group.default]
provisioner "local-exec" {
command = "mysql --host=${module.cluster.cluster_endpoint} --port=${module.cluster.cluster_port} --user=${module.cluster.cluster_master_username} --password=${module.cluster.cluster_master_password} --database=${module.cluster.cluster_database_name} < ${file(${path.module}/init/db_structure.sql)}"
}
}
(This snippet is based on this answer where you have a lot more examples)
Just note: Terraform manages infrastructure. When AWS provider does its work you can have MySQL provider to pick up and deploy admin stuff like users, roles, grants, etc. But tables within databases belong to application. There are other tools more suited for managing database objects. See if you could plug Flyway or Liquibase into your pipeline.

Deploying resource to multiple regions w/ TF 0.12/13

We have a rather complex environment where we have lots of AWS accounts, in multiple regions and these are all connected to a transit network via VPN tunnels.
At the moment we deploy Customer Gateways via a "VPC" module for each VPC in a region but the problem that we get is that deploying the first VPC is fine but subsequent VPC deploys cause issues with the fact that the CGW is already there and so we have to import it before we can continue which isn't an ideal place to be in and also I think there's a risk that if we tear down a VPC it might try to kill the CGW that is being used by other VPN's.
What I'm wanting to do is deploy the CGW's separately from the VPC and then the VPC does a data lookup for the CGW.
I've been thinking that perhaps we can use our "base" job to provision the CGW's that are defined in the variables file but nothing I've tried has worked so far.
The variable definition would be:
variable "region_data" {
type = list(object({
region = string
deploy_cgw = bool
gateways = any
}))
default = [
{
region = "eu-west-1"
deploy_cgw = true
gateways = [
{
name = "gateway1"
ip = "1.2.3.4"
},
{
name = "gateway2"
ip = "2.3.4.5"
}
]
},
{
region = "us-east-1"
deploy_cgw = true
gateways = [
{
name = "gateway1"
ip = "2.3.4.5"
},
{
name = "gateway2"
ip = "3.4.5.6"
}
]
}
]
}
I've tried a few things, like:
locals {
regions = [for region in var.region_data : region if region.deploy_cgw]
cgws = flatten([
for region in local.regions : [
for gateway in region.gateways : {
region = region.region
name = gateway.name
ip = gateway.ip
}
]
])
}
provider "aws" {
region = "eu-west-1"
alias = "eu-west-1"
}
provider "aws" {
region = "us-east-1"
alias = "us-east-1"
}
module "cgw" {
source = "../../../modules/customer-gateway"
for_each = { for cgw in local.cgws: "${cgw.region}.${cgw.name}" => cgw }
name_tag = each.value.name
ip_address = each.value.ip
providers = {
aws = "aws.${each.value.region}"
}
}
But with this I get:
Error: Invalid provider configuration reference
on main.tf line 439, in module "cgw":
439: aws = "aws.${each.value.region}"
A provider configuration reference must not be given in quotes.
If I move the AWS provider into the module and pass the region as a parameter, I get the following:
Error: Module does not support for_each
on main.tf line 423, in module "cgw":
423: for_each = { for cgw in local.testing : "${cgw.region}.${cgw.name}" => cgw }
Module "cgw" cannot be used with for_each because it contains a nested
provider configuration for "aws", at
I've done quite a bit of research and the last one I understand is something that Terraform take a tough stance on.
Is what I'm asking possible?
for_each can't be used on modules that have providers defined within them. I was disappointed to find this out too. They do this because having nested providers does cause nightmares if that provider goes away, then you have orphaned resources in the state that you can't manage and your plans will fail. It is, however, entirely possible in https://www.pulumi.com/. I'm sick of the limitations in terraform and will be moving to pulumi. But that's not what you asked so I'll move on.
Definitely don't keep importing it. You'll end up with multiple parts of your terraform managing the same resource.
Just create the cgw once per region. Then pass the id into your vpc module. You can't iterate over providers, so have one module per provider. In other words, for each over all vpcs in the same account and same region per module call.
resource "aws_customer_gateway" "east" {
bgp_asn = 65000
ip_address = "172.83.124.10"
type = "ipsec.1"
}
resource "aws_customer_gateway" "west" {
bgp_asn = 65000
ip_address = "172.83.128.10"
type = "ipsec.1"
}
module "east" {
source = "../../../modules/customer-gateway"
for_each = map(
{
name = "east1"
ip = "1.2.3.4"
},
{
name = "east2"
ip = "1.2.3.5"
},
)
name_tag = each.value.name
ip_address = each.value.ip
cgw_id = aws_customer_gateway.east.id
providers = {
aws = "aws.east"
}
}
module "west" {
source = "../../../modules/customer-gateway"
for_each = map(
{
name = "west1"
ip = "1.2.3.4"
},
{
name = "west2"
ip = "1.2.3.5"
},
)
name_tag = each.value.name
ip_address = each.value.ip
cgw_id = aws_customer_gateway.west.id
providers = {
aws = "aws.west"
}
}

Terraform - re-use an existing subnetwork to create a cloud sql instance on GCP

I am attempting to create a cloud sql instance on GCP using terraform. I want to use an existing VPC subnetwork created in an earlier step but there does not seem to be a way to refer to it. Instead all examples seem to require a new IP range to be setup. This is my current code that creates the new IP range:
provider = google-beta
project = "project_name"
name = "private_range"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 18
network = "projects/project_name/global/networks/vpc_name"
address = "192.168.128.0"
}
resource "google_service_networking_connection" "private_vpc_connection" {
provider = google-beta
network = "projects/project_name/global/networks/vpc_name"
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = [google_compute_global_address.private_ip_address.name]
}
resource "google_sql_database_instance" "instance" {
provider = google-beta
project = "project_name"
name = "db-instance10"
region = "us-east1"
database_version = "MYSQL_5_7"
depends_on = [google_service_networking_connection.private_vpc_connection]
settings {
tier = "db-f1-micro"
ip_configuration {
ipv4_enabled = false
private_network = "projects/project_name/global/networks/vpc_name"
}
}
}
provider "google-beta" {
region = "us-east1"
zone = "us-east1-c"
}
When I specify the exact same IP range as the existing subnet. I receive the error:
Error: Error waiting to create GlobalAddress: Error waiting for Creating GlobalAddress: Requested range conflicts with other resources: The provided IP range overlaps with existing subnetwork IP range.
There does not seem to be any obvious way to refer to the existing subnetwork resource as the reserved_peering_ranges parameter only seems to accept the name of an IP address range resource.
Here is the resource specification for the existing subnetwork:
creation_timestamp = "2020-06-03T07:28:05.762-07:00"
enable_flow_logs = true
fingerprint = "ied1TiEZjgc="
gateway_address = "192.168.128.1"
id = "us-east1/vpc_subnet_name"
ip_cidr_range = "192.168.128.0/18"
name = "vpc_subnet_name"
network = "https://www.googleapis.com/compute/v1/projects/project_name/global/networks/vpc_name"
private_ip_google_access = true
project = "project_name"
region = "us-east1"
secondary_ip_range = []
self_link = "https://www.googleapis.com/compute/v1/projects/project_name/regions/us-east1/subnetworks/vpc_subnet_name"
log_config {
aggregation_interval = "INTERVAL_5_SEC"
flow_sampling = 0.5
metadata = "INCLUDE_ALL_METADATA"
}
}
Connecting to a Cloud sql instance through a private IP requires configuring private service access that uses an allocated IP address range that must not overlap with any existing VPC subnet.
The private connection links your VPC network with the service's VPC network. This connection allows VM instances in your VPC network to use internal IP addresses to reach the service resources, for example a Cloud sql instance that has internal IP addresses.
Once created, the allocated IP address range and the connection can then be reused with other services.
you can use the below module to create the cloud sql with exisiting private vpc but you need to modify it according to your network. in this scenario i have created a separate private network & creating the cloud sql using that network.
https://github.com/gruntwork-io/terraform-google-sql
get the existing network in your cloud infra from which you want to create your cloudsql, the below command gives the
gcloud network list --uri
you need to append the network where self link is mentioned & hash out the steps where the vpc is getting created. please refer the below main.tf file
the location of this file is --- Cloud_SQL.terraform\modules\sql_example_postgres-private-ip\examples\postgres-private-ip\main.tf
add the variable accordingly.
# ------------------------------------------------------------------------------
# LAUNCH A POSTGRES CLOUD SQL PRIVATE IP INSTANCE
# ------------------------------------------------------------------------------
# ------------------------------------------------------------------------------
# CONFIGURE OUR GCP CONNECTION
# ------------------------------------------------------------------------------
provider "google-beta" {
project = var.project
region = var.region
}
terraform {
# This module is now only being tested with Terraform 0.14.x. However, to make upgrading easier, we are setting
# 0.12.26 as the minimum version, as that version added support for required_providers with source URLs, making it
# forwards compatible with 0.14.x code.
required_version = ">= 0.12.26"
required_providers {
google-beta = {
source = "hashicorp/google-beta"
version = "~> 3.57.0"
}
}
}
# ------------------------------------------------------------------------------
# CREATE A RANDOM SUFFIX AND PREPARE RESOURCE NAMES
# ------------------------------------------------------------------------------
resource "random_id" "name" {
byte_length = 2
}
####################################################################
# Reserve global internal address range for the peering
resource "google_compute_global_address" "private_ip_address" {
provider = google-beta
# name = local.private_ip_name
name = var.vpc_network
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 16
# network = google_compute_network.private_network.self_link
# network = google_compute_network.vpc_network.self_link
network = "https://www.googleapis.com/compute/v1/projects/lucky-operand-312611/global/networks/myprivatevpc/"
}
# Establish VPC network peering connection using the reserved address range
resource "google_service_networking_connection" "private_vpc_connection" {
provider = google-beta
# network = google_compute_network.private_network.self_link
network = "https://www.googleapis.com/compute/v1/projects/lucky-operand-312611/global/networks/myprivatevpc"
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = [google_compute_global_address.private_ip_address.name]
}
# ------------------------------------------------------------------------------
# CREATE DATABASE INSTANCE WITH PRIVATE IP
# ------------------------------------------------------------------------------
module "postgres" {
# When using these modules in your own templates, you will need to use a Git URL with a ref attribute that pins you
# to a specific version of the modules, such as the following example:
# source = "github.com/gruntwork-io/terraform-google-sql.git//modules/cloud-sql?ref=v0.2.0"
source = "../../modules/cloud-sql"
project = var.project
region = var.region
name = var.instance_name
db_name = var.db_name
engine = var.postgres_version
machine_type = var.machine_type
# To make it easier to test this example, we are disabling deletion protection so we can destroy the databases
# during the tests. By default, we recommend setting deletion_protection to true, to ensure database instances are
# not inadvertently destroyed.
deletion_protection = false
# These together will construct the master_user privileges, i.e.
# 'master_user_name'#'master_user_host' IDENTIFIED BY 'master_user_password'.
# These should typically be set as the environment variable TF_VAR_master_user_password, etc.
# so you don't check these into source control."
master_user_password = var.master_user_password
master_user_name = var.master_user_name
master_user_host = "%"
# Pass the private network link to the module
# private_network = google_compute_network.private_network.self_link
private_network = "https://www.googleapis.com/compute/v1/projects/lucky-operand-312611/global/networks/myprivatevpc"
# Wait for the vpc connection to complete
dependencies = [google_service_networking_connection.private_vpc_connection.network]
custom_labels = {
test-id = "postgres-private-ip-example"
}
}

How to fix "An Unknown Error Occurred" when creating multiple Google Cloud SQL instances with private IP simultaneously?

Our cloud backend setup contains 5 Cloud SQL for Postgres instances. We manage our infrastructure using Terraform. We are using connecting them from GKE using a public IP and the Cloud SQL container.
In order to simplify our setup we wish to get rid of the proxy containers by moving to a private IP. I tried following the Terraform guide. While a creating a single instance works fine, trying to create 5 instances simultaneously ends in 4 failed ones and one successful:
The error which appears in the Google Clod Console on the failed instances is "An Unknown Error occurred":
Following is the code which reproduces it. Pay attention to the count = 5 line:
resource "google_compute_network" "private_network" {
provider = "google-beta"
name = "private-network"
}
resource "google_compute_global_address" "private_ip_address" {
provider = "google-beta"
name = "private-ip-address"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 16
network = "${google_compute_network.private_network.self_link}"
}
resource "google_service_networking_connection" "private_vpc_connection" {
provider = "google-beta"
network = "${google_compute_network.private_network.self_link}"
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = ["${google_compute_global_address.private_ip_address.name}"]
}
resource "google_sql_database_instance" "instance" {
provider = "google-beta"
count = 5
name = "private-instance-${count.index}"
database_version = "POSTGRES_9_6"
depends_on = [
"google_service_networking_connection.private_vpc_connection"
]
settings {
tier = "db-custom-1-3840"
availability_type = "REGIONAL"
ip_configuration {
ipv4_enabled = "false"
private_network = "${google_compute_network.private_network.self_link}"
}
}
}
provider "google-beta" {
version = "~> 2.5"
credentials = "credentials.json"
project = "PROJECT_ID"
region = "us-central1"
zone = "us-central1-a"
}
I tried several alternatives:
Waiting a minute after creating the google_service_networking_connection and then creating all the instances simultaneously, but I got the same error.
Creating an address range and a google_service_networking_connection per instance, but I got an error that google_service_networking_connection cannot be created simultaneously.
Creating an address range per instance and a single google_service_networking_connection which links to all of them, but I got the same error.
Found an ugly yet working solution. There is a bug in GCP which does not prevent simultaneous creation of instances although it cannot be completed. There is neither documentation about it nor a meaningful error message. It appears in the Terraform Google provider issue tracker as well.
One alternative is adding a dependence between the instances. This allows their creation to complete successfully. However, each instance takes several minutes to create. This accumulates to many spent minutes. If we add an artificial delay of 60 seconds between instance creation, we manage to avoid the failures. Notes:
The needed amount of seconds to delay depends on the instance tier. For example, for db-f1-micro, 30 seconds were enough. They were not enough for db-custom-1-3840.
I am not sure what is the exact number of needed seconds for db-custom-1-3840. 30 seconds were not enough, 60 were.
Following is a the code sample to resolve the issue. It shows 2 instances only since due to depends_on limitations I could not use the count feature and showing the full code for 5 instances would be very long. It works the same for 5 instances:
resource "google_compute_network" "private_network" {
provider = "google-beta"
name = "private-network"
}
resource "google_compute_global_address" "private_ip_address" {
provider = "google-beta"
name = "private-ip-address"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 16
network = "${google_compute_network.private_network.self_link}"
}
resource "google_service_networking_connection" "private_vpc_connection" {
provider = "google-beta"
network = "${google_compute_network.private_network.self_link}"
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = ["${google_compute_global_address.private_ip_address.name}"]
}
locals {
db_instance_creation_delay_factor_seconds = 60
}
resource "null_resource" "delayer_1" {
depends_on = ["google_service_networking_connection.private_vpc_connection"]
provisioner "local-exec" {
command = "echo Gradual DB instance creation && sleep ${local.db_instance_creation_delay_factor_seconds * 0}"
}
}
resource "google_sql_database_instance" "instance_1" {
provider = "google-beta"
name = "private-instance-delayed-1"
database_version = "POSTGRES_9_6"
depends_on = [
"google_service_networking_connection.private_vpc_connection",
"null_resource.delayer_1"
]
settings {
tier = "db-custom-1-3840"
availability_type = "REGIONAL"
ip_configuration {
ipv4_enabled = "false"
private_network = "${google_compute_network.private_network.self_link}"
}
}
}
resource "null_resource" "delayer_2" {
depends_on = ["google_service_networking_connection.private_vpc_connection"]
provisioner "local-exec" {
command = "echo Gradual DB instance creation && sleep ${local.db_instance_creation_delay_factor_seconds * 1}"
}
}
resource "google_sql_database_instance" "instance_2" {
provider = "google-beta"
name = "private-instance-delayed-2"
database_version = "POSTGRES_9_6"
depends_on = [
"google_service_networking_connection.private_vpc_connection",
"null_resource.delayer_2"
]
settings {
tier = "db-custom-1-3840"
availability_type = "REGIONAL"
ip_configuration {
ipv4_enabled = "false"
private_network = "${google_compute_network.private_network.self_link}"
}
}
}
provider "google-beta" {
version = "~> 2.5"
credentials = "credentials.json"
project = "PROJECT_ID"
region = "us-central1"
zone = "us-central1-a"
}
provider "null" {
version = "~> 1.0"
}
In case someone lands here with a slightly different case (creating google_sql_database_instance in a private network results in an "Unknown error"):
Launch one Cloud SQL instance manually (this will enable servicenetworking.googleapis.com and some other APIs for the project it seems)
Run your manifest
Terminate the instance created in step 1.
Works for me after that
¯_(ツ)_/¯
I land here with a slightly different case, same as #Grigorash Vasilij
(creating google_sql_database_instance in a private network results in an "Unknown error").
I was using the UI to deploy an SQL instance on a private VPC, for some reason that trows me an "Unknown error" as well. I finally solved using the gcloud command instead (why that works and no the UI? IDK, maybe the UI is not doing the same as the command)
gcloud --project=[PROJECT_ID] beta sql instances create [INSTANCE_ID]
--network=[VPC_NETWORK_NAME]
--no-assign-ip
follow this for more details