Terraform output variable populates only during the second execution - amazon-web-services

Question

I am facing a weird situation while running terraform apply. While running terraform apply for the very first time I get the below error. Populated Terraform output is read in a separate python script so when api_gateway_id is null the python script fails.
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): Gathering data from Terraform...
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): Traceback (most recent call last):
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): File "/home/ec2-user/bin/lambda-deploy", line 65, in
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): provide_env('LAMBDA_GATEWAY_ID', 'api_gateway_id')
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): File "/home/ec2-user/bin/lambda-deploy", line 57, in provide_env
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): os.environ[key] = TERRAFORM_OUTPUTS[tf_output]
module.serverless_projects["api/admin.zip"].null_resource.serverless_deployment (local-exec): KeyError: 'api_gateway_id'
When I retry 'terraform apply' for the second time the value (api_gateway_id) is getting populated and execution is also successful.
main.tf
resource "aws_api_gateway_rest_api" "gw" {
name = "LambdaApiGateway"
endpoint_configuration {
types = ["PRIVATE"]
vpc_endpoint_ids = [aws_vpc_endpoint.gw.id]
}
binary_media_types = ["*/*"]
}
resource "aws_api_gateway_resource" "api" {
rest_api_id = aws_api_gateway_rest_api.gw.id
parent_id = aws_api_gateway_rest_api.gw.root_resource_id
path_part = "api"
}
Not only api_gateway_id, even api_gateway_root, api_gateway_resources are also not getting printed on first time execution. I tried terraform init, terraform refresh before the execution, but no luck.
output.tf
output "api_gateway_id" {
value = aws_api_gateway_rest_api.gw.id
}
output "api_gateway_root" {
value = aws_api_gateway_rest_api.gw.root_resource_id
}
output "api_gateway_resources" {
value = jsonencode({
api = aws_api_gateway_resource.api.id,
})
}
resource "null_resource" "serverless_deployment" {
triggers = {
source_version = data.aws_s3_bucket_object.package_object.version_id
}
provisioner "local-exec" {
command = "lambda-deploy ${var.package_name}"
}
}

Answer 1

Terraform only finalizes the updated state snapshot after the apply phase is complete, so it isn't reliable to try to access the state from actions taken during that apply phase.
Instead I would recommend passing the needed values directly to the external program you are running, either using command line arguments or using environment variables set only for that process.
For example:
provisioner "local-exec" {
command = "lambda-deploy ${var.package_name}"
environment = {
LAMBDA_GATEWAY_ID = aws_api_gateway_rest_api.gw.id
}
}
In the above I used an environment variable name that I saw mentioned in the stack trace your program produced, though you can use any environment variable names you wish and then access them in whatever is the normal way to do so inside the programming language you're using to implement this program. For a Python program, you will no longer need to assign them into os.environ, because they should already be available there.