Terraform does not update AWS canary code - amazon-web-services

Question

I have being changing an AWS canary code.
After running terraform apply, I see the updates in the new zip file but in AWS console the code is the old on.
What have I done wrong?
My terraform code:
resource "aws_synthetics_canary" "canary" {
depends_on = [time_sleep.wait_5_minutes]
name = var.name
artifact_s3_location = "s3://${local.artifacts_bucket_and_path}"
execution_role_arn = aws_iam_role.canary_role.arn
handler = "apiCanary.handler"
start_canary = true
zip_file = data.archive_file.source_zip.output_path
runtime_version = "syn-nodejs-puppeteer-3.3"
tags = {
Description = var.description
Entity = var.entity
Service = var.service
}
run_config {
timeout_in_seconds = 300
}
schedule {
expression = "rate(${var.rate_in_minutes} ${var.rate_in_minutes == 1 ? "minute" : "minutes"})"
}
}
I read this but it didn't help me.

Answer 1

I agree with #mjd2 but in the meantime worked around it by manually hashing the lambda source and embedding that hash into the source file name:
locals {
source_code = <whatever your source is>
source_code_hash = sha256(local.source_code)
}
data "archive_file" "canary_lambda" {
type = "zip"
output_path = "/tmp/canary_lambda_${local.source_code_hash}.zip"
source {
content = local.source_code
filename = "nodejs/node_modules/heartbeat.js"
}
}
This way, anytime the source_code is edited a new output filename will be used, triggering a replacement of the archive resource.

Answer 2

This could be a permission issue with your deployment role. Your role must have permission to modify the lambda behind the canary in order to apply the new layer that your zip file change creates.
Unfortunately any errors that occur when applying changes to the lambda are not communicated via terraform, or anywhere in the AWS console, but if it fails then your canary will continue to point to an old version of the lambda, without your code changes.
You should be able to see which version of the lambda your canary is using by checking the "Script location" field on the Configuration tab for your canary. Additionally, if you click in to the script location you will be able to see if you have newer, unpublished versions of the lambda layer available with your code changes in it.
To verify if the failure is a permission issue you need to query your canary via the AWS CLI.
Run aws synthetics get-canary --name <your canary name> and check the Status.StateReason.
If there was a permission issue when attempting to apply your change you should see something along the lines of:
<user> is not authorized to perform: lambda:UpdateFunctionConfiguration on resource: <lamdba arn>
Based on the above you should be able to add any missing permissions to your deployment roles iam policy and try your deployment again.

Answer 3

Hit the same issue. Seems like canary itself is a beta project that made it to production and the terraform resource that manages it also leaves much to be desired. There is no source_code_hash attr like with lambda, so you need to taint the entire canary resource so it gets recreated with any updated code. AWS Canary as of Nov 2022 is not mature at all. It should support integration with slack or at least AWS Chatbot out of the box, but it doesn't. Hopefully AWS team gives it some love because it's terrible as is right now in comparison to NewRelic, Dynatrace, and most other monitoring services that support synthetics.