Terraform update only a cloud function from a bunch - google-cloud-platform

I have a Terraform project that allows to create multiple cloud functions.
I know that if I change the name of the google_storage_bucket_object related to the function itself, terraform will see the difference of the zip name and redeploy the cloud function.
My question is, there is a way to obtain the same behaviour, but only with the cloud functions that have been changed?
resource "google_storage_bucket_object" "zip_file" {
# Append file MD5 to force bucket to be recreated
name = "${local.filename}#${data.archive_file.source.output_md5}"
bucket = var.bucket.name
source = data.archive_file.source.output_path
}
# Create Java Cloud Function
resource "google_cloudfunctions_function" "java_function" {
name = var.function_name
runtime = var.runtime
available_memory_mb = var.memory
source_archive_bucket = var.bucket.name
source_archive_object = google_storage_bucket_object.zip_file.name
timeout = 120
entry_point = var.function_entry_point
event_trigger {
event_type = var.event_trigger.event_type
resource = var.event_trigger.resource
}
environment_variables = {
PROJECT_ID = var.env_project_id
SECRET_MAIL_PASSWORD = var.env_mail_password
}
timeouts {
create = "60m"
}
}
By appending MD5 every cloud functions will result in a different zip file name, so terraform will re-deploy every of them and I found that without the MD5, Terraform will not see any changes to deploy.
If I have changed some code only inside a function, how can I tell to Terraform to re-deploy only it (so for example to change only its zip file name)?
I hope my question is clear and I want to thank you everyone who tries to help me!

Related

issue while deploying gcp cloud function deployment

I have following issue while deploying the cloud function (I am completely new to gcp and terraform )
I am trying to deploy a cloud function through the terraform; but the issue is that when I am deploying its destroying an existing cloud function which was already deployed in gcp (deployed by other colleague) even though cloud function name , bucket object name and archive file name are different (only bucket name and project id are same)
looks like its taking the state of existing cloud function which is already deployed
is there any way to keep the existing state unaffected?
code Snippet(as mentioned above there is already one cloud function deployed with same project id and bucket)
main.tf:
provider "google" {
project = "peoject_id"
credentials = "cdenetialfile"
region = "some-region"
}
locals {
timestamp = formatdate("YYMMDDhhmmss", timestamp())
root_dir = abspath("./app/")
}
data "archive_file" "archive" {
type = "zip"
output_path = "/tmp/function-${local.timestamp}.zip"
source_dir = local.root_dir
}
resource "google_storage_bucket_object" "object_archive" {
name = "archive-${local.timestamp}.zip"
bucket = "dev-bucket-tfstate"
source = "/tmp/function-${local.timestamp}.zip"
depends_on = [data.archive_file.archive]
}
resource "google_cloudfunctions_function" "translator_function" {
name = "Cloud_functionname"
available_memory_mb = 256
timeout = 61
runtime = "java11"
source_archive_bucket = "dev-bucket-tfstate"
source_archive_object = google_storage_bucket_object.object_archive.name
entry_point = "com.test.controller.myController"
event_trigger {
event_type = "google.pubsub.topic.publish"
resource = "topic_name"
}
}
backend.tf
terraform {
backend "gcs" {
bucket = "dev-bucket-tfstate"
credentials = "cdenetialfile"
}
}

Terraform GCP executes resources in wrong order

I have this main.tf file:
provider "google" {
project = var.projNumber
region = var.regName
zone = var.zoneName
}
resource "google_storage_bucket" "bucket_for_python_application" {
name = "python_bucket_exam"
location = var.regName
force_destroy = true
}
resource "google_storage_bucket_object" "file-hello-py" {
name = "src/hello.py"
source = "app-files/src/hello.py"
bucket = "python_bucket_exam"
}
resource "google_storage_bucket_object" "file-main-py" {
name = "main.py"
source = "app-files/main.py"
bucket = "python_bucket_exam"
}
When executed first time It worked fine, but after terraform destroy and again terraform plan -> terraform apply I've noticed that terraform tries to create object before actually creating a bucket:
Ofc it cant't create object inside something that does'nt exist. Why is that?
You have to create a dependency between your objects and your bucket (see code below). Otherwise, Terraform won't know that it has to create bucket first, and then objects. This is related to how Terraform stores the resources in a directed graph.
resource "google_storage_bucket_object" "file-hello-py" {
name = "src/hello.py"
source = "app-files/src/hello.py"
bucket = google_storage_bucket.bucket_for_python_application.name
}
resource "google_storage_bucket_object" "file-main-py" {
name = "main.py"
source = "app-files/main.py"
bucket = google_storage_bucket.bucket_for_python_application.name
}
By doing this, you declare an implicit order : bucket, then objects. This is equivalent to using depends_on in your google_storage_bucket_objects, but in that particular case I recommend using a reference to your bucket in your objects, rather than using an explicit depends_on.

Terraform: How to pass output from one resource to another?

I'm using Aurora serverless Mysql and ECS and trying to use secrets generated by aws secret manager in a file named rds.tf and want to use it another resource in a file called ecs.tf
resource "random_password" "db_instance_aurora_password" {
length = 40
special = false
keepers = {
database_id = aws_secretsmanager_secret.db_instance_aurora_master_password.id
}
Above is rds.tf, which works and generates a random password. In my second file ecs.tf, I want to use the
resource "aws_ecs_task_definition" "task" {
family = var.service_name
container_definitions = templatefile("${path.module}/templates/task_definition.tpl", {
DB_USERNAME = var.db_username
DB_PASSWORD = random_password.db_instance_aurora_password.result
})
}
How to export, the output of the db_password and use it in another resource(ecs.tf)?
output "aurora_rds_cluster.master_password" {
description = "The master password"
value = random_password.db_instance_aurora_password.result }
If all terraform files are in one directory, you can just reference random_password resource as you do it for the database. Then you might not need to output it.
If it's separated, then you can use terraform modules to achieve what you need. In ECS terraform you can reference RDS module and you will have access to its output:
module "rds" {
source = "path/to/folder/with/rds/terraform"
}
resource "aws_ecs_task_definition" "task" {
family = var.service_name
container_definitions = templatefile("${path.module}/templates/task_definition.tpl", {
DB_USERNAME = var.db_username
DB_PASSWORD = module.rds.aurora_rds_cluster.master_password
})
}
Storing password in terraform's output will store it as a plain text. Even if you use encrypted S3 bucket, password can still be accessed at least by terraform. Another option to share password could be for example by using AWS Parameter Store. Module that creates password can store it in Param Store, and another module that needs a password can read it.
P.S. You might want to add sensitive = true to the password output in order to eliminate password value from logs.

Terraform not uploading a new ZIP

I want to use Terraform for deployment of my lambda functions. I did something like:
provider "aws" {
region = "ap-southeast-1"
}
data "archive_file" "lambda_zip" {
type = "zip"
source_dir = "src"
output_path = "build/lambdas.zip"
}
resource "aws_lambda_function" "test_terraform_function" {
filename = "build/lambdas.zip"
function_name = "test_terraform_function"
handler = "test.handler"
runtime = "nodejs8.10"
role = "arn:aws:iam::000000000:role/xxx-lambda-basic"
memory_size = 128
timeout = 5
source_code_hash = "${data.archive_file.lambda_zip.output_base64sha256}"
tags = {
"Cost Center" = "Consulting"
Developer = "Jiew Meng"
}
}
I find that when there is no change to test.js, terraform correctly detects no change
No changes. Infrastructure is up-to-date.
When I do change the test.js file, terraform does detect a change:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
~ aws_lambda_function.test_terraform_function
last_modified: "2018-12-20T07:47:16.888+0000" => <computed>
source_code_hash: "KpnhsytFF0yul6iESDCXiD2jl/LI9dv56SIJnwEi/hY=" => "JWIYsT8SszUjKEe1aVDY/ZWBVfrZYhhb1GrJL26rYdI="
It does zip up the new zip, however, it does not seem to update the function with the new ZIP. It seems like it thinks since the filename has no change, it does not upload ... How can I fix this behaviour?
=====
Following some of the answers here, I tried:
Using null_resource
Using S3 bucket/object with etag
And it does not update ... Why is that?
I ran into the same issue and what solved it for me was publishing the Lambda functions automatically using the publish argument. To do so simply set publish = true in your aws_lambda_function resource.
Note that your function will be versioned after this and each change will create a new one. Therefor you should make sure that you use the qualified_arn attribute reference if you're referring to the function in any of your other Terraform code.
There is a workaround to trigger the resource to be refreshed, if the target lambda file names are src/main.py and src/handler.py. If you have more files to be managed, add them one by one.
resource "null_resource" "lambda" {
triggers {
main = "${base64sha256(file("src/main.py"))}"
handler = "${base64sha256(file("src/handler.py"))}"
}
}
data "archive_file" "lambda_zip" {
type = "zip"
source_dir = "src"
output_path = "build/lambdas.zip"
depends_on = ["null_resource.lambda"]
}
Let me know if this works for you.
There is 2 things you need to take care of:
upload zip file to S3 if its content has changed
update Lambda function if zip file content has changed
I can see you are taking care of the latter with source_code_hash. I don't see how you handle the former. It could look like that:
resource "aws_s3_bucket_object" "zip" {
bucket = "${aws_s3_bucket.zip.bucket}"
key = "myzip.zip"
source = "${path.module}/myzip.zip"
etag = "${md5(file("${path.module}/myzip.zip"))}"
}
etag is the most important option here.
I created this module to help ease some of the issues around deploying Lambda with Terraform: https://registry.terraform.io/modules/rojopolis/lambda-python-archive/aws/0.1.4
It may be useful in this scenario. Basically, it replaces the "archive_file" data source with a specialized lambda archive data source to better manage stable source code hash, etc.

Having trouble with Terraform and AWS Storage Gateway disks

I am using Terraform with AWS and have been able to create an AWS Storage Gateway file gateway using the aws_storagegateway_gateway resource.
The gateway will create and the status will be 'online' however there is not a cache disk added yet in the console which is normal as it has to be done after the gateway is created. The VM does have a disk and it is available to add in the console and doing so in the console works perfectly.
However, I am trying to add the disk with Terraform once the gateway is created and cannot seem to get the code to work, or quite possibly don't understand how to get it to work.
Trying to use the aws_storagegateway_cache resource but I get an error on the disk_id and do not know how to get it to return from the code of the gateway creation.
Might someone have a working example of how to get the cache disk to add with Terraform once the gateway is created or know how to get the disk_id so I can add it?
Adding code
provider "aws" {
access_key = "${var.access-key}"
secret_key = "${var.secret-key}"
token = "${var.token}"
region = "${var.region}"
}
resource "aws_storagegateway_gateway" "hmsgw" {
gateway_ip_address = "${var.gateway-ip-address}"
gateway_name = "${var.gateway-name}"
gateway_timezone = "${var.gateway-timezone}"
gateway_type = "${var.gateway-type}"
smb_active_directory_settings {
domain_name = "${var.domain-name}"
username = "${var.username}"
password = "${var.password}"
}
}
resource "aws_storagegateway_cache" "sgwdisk" {
disk_id = "SCSI"
gateway_arn = "${aws_storagegateway_gateway.hmsgw.arn}"
}
output "gatewayid" {
value = "${aws_storagegateway_gateway.hmsgw.arn}"
}
The error I get is:
aws_storagegateway_cache.sgwdisk: error adding Storage Gateway cache: InvalidGatewayRequestException: The specified disk does not exist.
status code: 400, request id: fda602fd-a47e-11e8-a1f4-b383e2e2e2f6
I have attempted to hard code the disk_id like above or use a variable. On the variable I don't know if it is returned or exists so that could be the issue, new to this.
Before creating the resource "aws_storagegateway_cache", use data to get the disk id. I am using the below scripts and it works fine.
variable "upload_disk_path" {
default = "/dev/sdb"
}
data "aws_storagegateway_local_disk" "upload_disk" {
disk_path = "${var.upload_disk_path}"
gateway_arn = "${aws_storagegateway_gateway.this.arn}"
}
resource "aws_storagegateway_upload_buffer" "stg_upload_buffer" {
disk_id = "${data.aws_storagegateway_local_disk.upload_disk.disk_id}"
gateway_arn = "${aws_storagegateway_gateway.this.arn}"
}
In case your are using two disk's (one for upload and one cahce), use the same code but set the default value of cache_disk_path = "/dev/sdc"
if you use the AWS cli to run: aws storagegateway list-local-disks --gateway-arn [your gateway's arn] --region [gateway's region], you'll get data returned that includes the disk ID.
Then, in your example code, you replace SCSI with "${gateway_arn}:[diskID from command above]" and your cache volume will be created.
One thing I've noticed though - when I've done this and then tried to apply the same Terraform code again, and in some cases even with a targeted deploy of a specific resource within my Terraform, it wants to re-deploy the cache volume, because the Terraform is detecting that the disk ID is changing to a value of "1". Passing in "1" as the value in the Terraform, however, does not seem to work.
This would work also:
variable "disk_path" {
default = "/dev/sdb"
}
provider "aws" {
alias = "primary"
access_key = var.access-key
secret_key = var.secret-key
token = var.token
region = var.region
}
resource "aws_storagegateway_gateway" "hmsgw" {
gateway_ip_address = var.gateway-ip-address
gateway_name = var.gateway-name
gateway_timezone = var.gateway-timezone
gateway_type = var.gateway-type
smb_active_directory_settings {
domain_name = var.domain-name
username = var.username
password = var.password
}
}
data "aws_storagegateway_local_disk" "sgw_disk" {
disk_path = var.disk_path
gateway_arn = aws_storagegateway_gateway.hmsgw.arn
provider = aws.primary
}
resource "aws_storagegateway_cache" "sgw_cache" {
disk_id = data.aws_storagegateway_local_disk.sgw_disk.id
gateway_arn = aws_storagegateway_gateway.hmsgw.arn
provider = aws.primary
}