Enable CloudWatch logs for AWS API Gateway using Terraform - amazon-web-services

I am using OpenAPI 3.0 spec to deploy an AWS API Gateway. I am not able to figure out how to enable cloud watch logs for the deployment.
Here is the terraform code:
data "template_file" "test_api_swagger" {
template = file(var.api_spec_path)
vars = {
//ommitted
}
}
resource "aws_api_gateway_rest_api" "test_api_gateway" {
name = "test_backend_api_gateway"
description = "API Gateway for some x"
body = data.template_file.test_api_swagger.rendered
endpoint_configuration {
types = ["REGIONAL"]
}
}
resource "aws_api_gateway_deployment" "test_lambda_gateway" {
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = var.env
}
I checked Amazon OpenAPI extensions and none seem to have this option. Only way I see is using api_gateway_method_settings which I cannot use in this case.

I think that it is not supported in terraform. I'm currently using terraform provisioner to run aws cli command after the deployment is created, like in the example below:
The example that I'm providing is to enable XRay tracing. You'll need to research the correct path and value to be used for CloudWatch logs. You can find more information in the docs.
resource "aws_api_gateway_deployment" "test_lambda_gateway" {
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = var.env
provisioner "local-exec" {
command = "aws apigateway update-stage --region ${data.aws_region.current.name} --rest-api-id ${aws_api_gateway_rest_api.test_api_gateway.id} --stage-name ${var.env} --patch-operations op=replace,path=/tracingEnabled,value=true"
}
}
You just need to make a reference to the aws data provider in your terraform template:
data "aws_region" "current" {}

Even though you're creating the gateway with OpenAPI import, you can still use api_gateway_method_settings to reference the stage, assuming you're using a stage as recommended. See AWS documentation. You would just indicate "*/*" on the method_path as per the example.
resource "aws_api_gateway_stage" "example" {
deployment_id = aws_api_gateway_deployment.test_lambda_gateway.id
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = "example"
}
resource "aws_api_gateway_method_settings" "all" {
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = aws_api_gateway_stage.example.stage_name
method_path = "*/*"
settings {
logging_level = "INFO"
}
}
This should set up the logging on the gateway for all requests with INFO level logging as if you had done it in the console on the stage.

Related

GCP API gateway returning 403 saying managed service "is not enabled for the project"

Trying to access a public cloud run service and not sure why I keep getting this error message ({"message":"PERMISSION_DENIED:API basic-express-api-1yy1jgrw4nwy2.apigateway.chrome-courage-336400.cloud.goog is not enabled for the project.","code":403}) when hitting the gateway default hostname path with the API key in query string. The config has a service account with the role to be able to invoke cloud run services. All required APIs are also enabled. Here is a link to my entire codebase, but below is my API Gateway specific terraform configuration.
resource "google_api_gateway_api" "basic_express" {
depends_on = [google_project_service.api_gateway, google_project_service.service_management, google_project_service.service_control]
provider = google-beta
api_id = "basic-express-api"
}
resource "google_api_gateway_api_config" "basic_express" {
depends_on = [google_project_service.api_gateway, google_project_service.service_management, google_project_service.service_control, google_api_gateway_api.basic_express]
provider = google-beta
api = google_api_gateway_api.basic_express.api_id
api_config_id = "basic-express-cfg"
openapi_documents {
document {
path = "api-configs/openapi-spec-basic-express.yaml"
contents = filebase64("api-configs/openapi-spec-basic-express.yaml")
}
}
lifecycle {
create_before_destroy = true
}
gateway_config {
backend_config {
google_service_account = google_service_account.apig_gateway_basic_express_sa.email
}
# https://cloud.google.com/api-gateway/docs/configure-dev-env?&_ga=2.177696806.-2072560867.1640626239#configuring_a_service_account
# when I added this terraform said that the resource already exists, so I had to tear down all infrastructure and re-provision - also did not make a difference, still getting a 404 error when trying to hit the gateway default hostname endpoint - this resource might be immutable...
}
}
resource "google_api_gateway_gateway" "basic_express" {
depends_on = [google_project_service.api_gateway, google_project_service.service_management, google_project_service.service_control, google_api_gateway_api_config.basic_express, google_api_gateway_api.basic_express]
provider = google-beta
api_config = google_api_gateway_api_config.basic_express.id
gateway_id = "basic-express-gw"
region = var.region
}
resource "google_service_account" "apig_gateway_basic_express_sa" {
account_id = "apig-gateway-basic-express-sa"
depends_on = [google_project_service.iam]
}
# "Identity to be used by gateway"
resource "google_project_iam_binding" "project" {
project = var.project_id
role = "roles/run.invoker"
members = [
"serviceAccount:${google_service_account.apig_gateway_basic_express_sa.email}"
]
}
# https://cloud.google.com/api-gateway/docs/configure-dev-env?&_ga=2.177696806.-2072560867.1640626239#configuring_a_service_account
Try:
PROJECT=[[YOUR-PROJECT]]
SERVICE="basic-express-api-1yy1jgrw4nwy2.apigateway.chrome-courage-336400.cloud.goog"
gcloud services enable ${SERVICE} \
--project=${PROJECT}
As others have pointed out, you need to enable the api service. You can do via terraform with the google_project_service resource:
resource "google_project_service" "basic_express" {
project = var.project_id
service = google_api_gateway_api.basic_express.managed_service
timeouts {
create = "30m"
update = "40m"
}
disable_dependent_services = true
}

How to refresh AWS Lambda permission for API Gateway using Terraform?

I am deploying a REST API Gateway using Terraform. Couple of endpoints are accessing Lambda function to return response. Whenever I deploy api-gw using terraform, the Lambda permission doesn't seem to refresh and I have to manually open the api-gw portal in AWS console and again add that lambda function post which it prompts me to allow invoke action. How can I refresh the permission without having to do these manual steps ? I am using below snippet for api-gw deployment and lambda permissions:
resource "aws_api_gateway_deployment" "deploy" {
rest_api_id = aws_api_gateway_rest_api.apigw.id
stage_name = ""
variables = {
deployed_at = timestamp()
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_lambda_permission" "customers_lambda_permission" {
statement_id = "AllowDemoAPIInvokeProjectGet"
action = "lambda:InvokeFunction"
function_name = local.lambda_name
principal = "apigateway.amazonaws.com"
source_arn = "${aws_api_gateway_rest_api.apigw.execution_arn}/*/GET/api/customers"
}
Your aws_api_gateway_deployment resource should depend on the aws_api_gateway_integration so that the lambda integration is created before deployment.
resource "aws_api_gateway_deployment" "deploy" {
...
depends_on = [
aws_api_gateway_integration.example1,
aws_api_gateway_integration.example2
]
}
or use triggers attribute:
resource "aws_api_gateway_deployment" "deploy" {
...
triggers = {
redeployment = sha1(jsonencode([
aws_api_gateway_resource.example1.id,
aws_api_gateway_method.example1.id,
aws_api_gateway_integration.example1.id,
]))
}

How to Configure CloudWatch Lambda Insights in Terraform

I need to enable "CloudWatch Lambda Insights" for a lambda using Terraform, but could not find the documentation. How I can do it in Terraform?
Note: This question How to add CloudWatch Lambda Insights to serverless config? may be relevant.
There is no "boolean switch" in the aws_lambda_function resource of the AWS Terraform provider that you can set to true, that would enable Cloudwatch Lambda Insights.
Fortunately, it is possible to do this yourself. The following Terraform definitions are based on this AWS documentation: Using the AWS CLI to enable Lambda Insights on an existing Lambda function
The process involves two steps:
Add a layer to your Lambda
Attach a AWS policy to your Lambdas role.
The Terraform definitions would look like this:
resource "aws_lambda_function" "insights_example" {
[...]
layers = [
"arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension:14"
]
}
resource "aws_iam_role_policy_attachment" "insights_policy" {
role = aws_iam_role.insights_example.id
policy_arn = "arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy"
}
Important: The arn of the layer is different for each region. The documentation I linked above has a link to a list of them. Furthermore, there is an additional step required if your Lambda is in a VPC, which you can read about in the documentation. The described "VPC step" can be put into Terraform as well.
For future readers: The version of that layer in my example is 14. This will change over time. So please do not just copy & paste that part. Follow the provided links and look for the current version of that layer.
Minimal, Complete, and Verifiable example
Tested with:
Terraform v0.14.4
+ provider registry.terraform.io/hashicorp/archive v2.0.0
+ provider registry.terraform.io/hashicorp/aws v3.24.0
Create the following two files (handler.py and main.tf) in a folder. Then run the following commands:
terraform init
terraform plan
terraform apply
Besides deploying the required resources, it will also create a zip archive containing the handler.py which is the deployment artifact used by the aws_lambda_function resource. So this is an all-in-one example without the need of further zipping etc.
handler.py
def lambda_handler(event, context):
return {
'message' : 'CloudWatch Lambda Insights Example'
}
main.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
provider "aws" {
region = "us-east-1"
}
resource "aws_lambda_function" "insights_example" {
function_name = "insights-example"
runtime = "python3.8"
handler = "handler.lambda_handler"
role = aws_iam_role.insights_example.arn
filename = "${path.module}/lambda.zip"
layers = [
"arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension:14"
]
depends_on = [
data.archive_file.insights_example
]
}
resource "aws_iam_role" "insights_example" {
name = "InsightsExampleLambdaRole"
assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}
resource "aws_iam_role_policy_attachment" "insights_example" {
role = aws_iam_role.insights_example.id
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}
resource "aws_iam_role_policy_attachment" "insights_policy" {
role = aws_iam_role.insights_example.id
policy_arn = "arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy"
}
data "aws_iam_policy_document" "lambda_assume" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["lambda.amazonaws.com"]
}
}
}
data "archive_file" "insights_example" {
type = "zip"
source_file = "${path.module}/handler.py"
output_path = "${path.module}/lambda.zip"
}
In case you are using container images as the deployment package for your Lambda function, the required steps to enable CloudWatch Lambda Insights are slightly different (since Lambda layers can't be used here):
attach the arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy to your functions role as described by Jens
Add the Lambda Insights extension to your container image
FROM public.ecr.aws/lambda/nodejs:12
RUN curl -O https://lambda-insights-extension.s3-ap-northeast-1.amazonaws.com/amazon_linux/lambda-insights-extension.rpm && \
rpm -U lambda-insights-extension.rpm && \
rm -f lambda-insights-extension.rpm
COPY app.js /var/task/
see documentation for details
Based off #jens' answer, here's a snippet that will automatically supply the correct LambdaInsightsExtension layer based on the current region:
data "aws_region" "current" {}
locals {
aws_region = data.aws_region.current.name
# List taken from https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-extension-versionsx86-64.html
lambdaInsightsLayers = {
"us-east-1" : "arn:aws:lambda:us-east-1:580247275435:layer:LambdaInsightsExtension:18",
"us-east-2" : "arn:aws:lambda:us-east-2:580247275435:layer:LambdaInsightsExtension:18",
"us-west-1" : "arn:aws:lambda:us-west-1:580247275435:layer:LambdaInsightsExtension:18",
"us-west-2" : "arn:aws:lambda:us-west-2:580247275435:layer:LambdaInsightsExtension:18",
"af-south-1" : "arn:aws:lambda:af-south-1:012438385374:layer:LambdaInsightsExtension:11",
"ap-east-1" : "arn:aws:lambda:ap-east-1:519774774795:layer:LambdaInsightsExtension:11",
"ap-south-1" : "arn:aws:lambda:ap-south-1:580247275435:layer:LambdaInsightsExtension:18",
"ap-northeast-3" : "arn:aws:lambda:ap-northeast-3:194566237122:layer:LambdaInsightsExtension:1",
"ap-northeast-2" : "arn:aws:lambda:ap-northeast-2:580247275435:layer:LambdaInsightsExtension:18",
"ap-southeast-1" : "arn:aws:lambda:ap-southeast-1:580247275435:layer:LambdaInsightsExtension:18",
"ap-southeast-2" : "arn:aws:lambda:ap-southeast-2:580247275435:layer:LambdaInsightsExtension:18",
"ap-northeast-1" : "arn:aws:lambda:ap-northeast-1:580247275435:layer:LambdaInsightsExtension:25",
"ca-central-1" : "arn:aws:lambda:ca-central-1:580247275435:layer:LambdaInsightsExtension:18",
"eu-central-1" : "arn:aws:lambda:eu-central-1:580247275435:layer:LambdaInsightsExtension:18",
"eu-west-1" : "arn:aws:lambda:eu-west-1:580247275435:layer:LambdaInsightsExtension:18",
"eu-west-2" : "arn:aws:lambda:eu-west-2:580247275435:layer:LambdaInsightsExtension:18",
"eu-south-1" : "arn:aws:lambda:eu-south-1:339249233099:layer:LambdaInsightsExtension:11",
"eu-west-3" : "arn:aws:lambda:eu-west-3:580247275435:layer:LambdaInsightsExtension:18",
"eu-north-1" : "arn:aws:lambda:eu-north-1:580247275435:layer:LambdaInsightsExtension:18",
"me-south-1" : "arn:aws:lambda:me-south-1:285320876703:layer:LambdaInsightsExtension:11",
"sa-east-1" : "arn:aws:lambda:sa-east-1:580247275435:layer:LambdaInsightsExtension:18"
}
}
resource "aws_lambda_function" "my_lambda" {
...
layers = [
local.lambdaInsightsLayers[local.aws_region]
]
}
resource "aws_iam_role_policy_attachment" "insights_policy" {
role = aws_iam_role.my_lambda.id
policy_arn = "arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy"
}

Why is the method response of an API gateway different when being created using terraform?

I have the following terraform script which creates an API gateway that passes requests to a lambda function.
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
#
region = "${var.region}"
version = "~> 2.6"
}
resource "aws_api_gateway_rest_api" "MyDemoAPI" {
name = "MyDemoAPI"
description = "This is my API for demonstration purposes"
}
resource "aws_api_gateway_resource" "MyDemoResource" {
rest_api_id = "${aws_api_gateway_rest_api.MyDemoAPI.id}"
parent_id = "${aws_api_gateway_rest_api.MyDemoAPI.root_resource_id}"
path_part = "mydemoresource"
}
resource "aws_api_gateway_method" "MyDemoMethod" {
rest_api_id = "${aws_api_gateway_rest_api.MyDemoAPI.id}"
resource_id = "${aws_api_gateway_resource.MyDemoResource.id}"
http_method = "POST"
authorization = "NONE"
}
resource "aws_api_gateway_integration" "MyDemoIntegration" {
rest_api_id = "${aws_api_gateway_rest_api.MyDemoAPI.id}"
resource_id = "${aws_api_gateway_resource.MyDemoResource.id}"
http_method = "${aws_api_gateway_method.MyDemoMethod.http_method}"
integration_http_method = "POST"
type = "AWS_PROXY"
uri = "arn:aws:apigateway:ap-southeast-1:lambda:path/2015-03-31/functions/${aws_lambda_function.test_lambda_function.arn}/invocations"
content_handling = "CONVERT_TO_TEXT"
}
resource "aws_api_gateway_method_response" "200" {
rest_api_id = "${aws_api_gateway_rest_api.MyDemoAPI.id}"
resource_id = "${aws_api_gateway_resource.MyDemoResource.id}"
http_method = "${aws_api_gateway_method.MyDemoMethod.http_method}"
status_code = "200"
response_models {
"application/json" = "Empty"
}
}
resource "aws_lambda_function" "test_lambda_function" {
filename = "lambda.zip"
description = "test build api gateway and lambda function using terraform"
function_name = "test_lambda_function"
role = "arn:aws:iam::123456789123:role/my_labmda_role"
handler = "gateway.lambda_handler"
runtime = "python3.6"
memory_size = 128
timeout = 60
}
The Method Response section of the API gateway resource display Select an integration response..
But if I create the same API gateway using AWS console, the Method Response section displays something different:
Why does this happen?
The following steps are how I use AWS console to create the API gateway:
Select Create Method under the resource.
Select POST method.
Select the desired options.
I've tried creating the above resources manually first, then execute terraform apply. Then terraform tells me that nothing needs to be changed.
terraform apply
aws_api_gateway_rest_api.MyDemoAPI: Refreshing state... (ID: 1qa34vs1k7)
aws_lambda_function.test_lambda_function: Refreshing state... (ID: test_lambda_function)
aws_api_gateway_resource.MyDemoResource: Refreshing state... (ID: 4xej81)
aws_api_gateway_method.MyDemoMethod: Refreshing state... (ID: agm-1qa34vs1k7-4xej81-POST)
aws_api_gateway_method_response.200: Refreshing state... (ID: agmr-1qa34vs1k7-4xej81-POST-200)
aws_api_gateway_integration.MyDemoIntegration: Refreshing state... (ID: agi-1qa34vs1k7-4xej81-POST)
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
This seems to mean that the manually built structure is the same as the structure built by terraform.
Because API Gateway is a complex AWS component and you can control pretty much everything on it (basically every single part of it is managed independently, giving you a lot of control over what you create but also making things harder to deal with).
See that it says "Select an Integration Response", but since your Terraform code didn't create one, it is therefore empty.
I had come across this very same problem a few weeks ago and I found the solution on Terraform's GitHub. I think Terraform should better document this as you're not the first one nor will you be the last to come up with this question. Well, at least we have this documented in StackOverflow now :)
Long story short, you need to add a aws_api_gateway_integration_response terraform resource to your API Gateway.
resource "aws_api_gateway_integration_response" "MyDemoIntegrationResponse" {
rest_api_id = "${aws_api_gateway_rest_api.MyDemoAPI.id}"
resource_id = "${aws_api_gateway_resource.MyDemoResource.id}"
http_method = "${aws_api_gateway_method.MyDemoMethod.http_method}"
status_code = "${aws_api_gateway_method_response.200.status_code}"
response_templates = {
"application/json" = ""
}
}
If you can, however, I suggest you use a proper framework to hook events to your Lambda functions (like the Serverless Framework or AWS SAM) as it's very verbose and error prone to create them in Terraform.
Usually, I combine Terraform and Serverless Framework together: I use Terraform to create infrastructure resources - even if they are serverless - like DynamoDB tables, SQS queues, SNS topics, etc. and the Serverless Framework to create the Lambda functions and their corresponding events.
I had the same issue. What Thales suggested didn't work for me (using terraform 0.11.14 with AWS plugin 2.52.0).
Instead, I used api_gateway_method_response resource and set the response_models parameter as:
response_models = {
"application/json" = "Empty"
}
Then it showed me the default response that all other APIs that I created through the console show!
Not very clear in terraform docs what this should look like, had to figure out myself after many attempts.

How to assign tags on an AWS API Gateway deployment stage using Terraform

I am using Hashicorp Terraform to define an AWS API Gateway to hit a Lambda function. I have a requirement that I need to tag my AWS resources with a particular tag so that costs can be tracked. Terraform seems to allow this for most resources. However, when creating an API Gateway stage using aws_api_gateway_deployment I do not have the option to specify tags.
I see that Terraform recently added the resource aws_api_gateway_stage. This one does allow tags to be specified. But, aws_api_gateway_stage requires an aws_api_gateway_deployment. If I give them the same "stage_name" as so:
resource "aws_api_gateway_stage" "PlayLambdaApiGatewayStage" {
stage_name = "${environment}"
rest_api_id = "${aws_api_gateway_rest_api.PlayLambdaApiGateway.id}"
deployment_id = "${aws_api_gateway_deployment.PlayLambdaApiGatewayDeployment.id}"
tags = {
cost-allocation = "play-${var.environment}"
}
}
resource "aws_api_gateway_deployment" "PlayLambdaApiGatewayDeployment" {
depends_on = [
"aws_api_gateway_integration.PlayLambdaApiLambdaIntegration",
"aws_api_gateway_integration.PlayLambdaApiLambdaIntegrationRoot"
]
rest_api_id = "${aws_api_gateway_rest_api.PlayLambdaApiGateway.id}"
stage_name = "${var.environment}"
}
Then they both resources try to create the stage and I get an error:
aws_api_gateway_stage.PlayLambdaApiGatewayStage: Error creating API Gateway Stage: ConflictException: Stage already exists
status code: 409, request id: f67a10c4-8aad-11e8-b486-c337ea2d214f
Here it would seem that the aws_api_gateway_deployment already created the stage, so the aws_api_gateway_stage resource failed to create it also. If I add the stage to the deployment's "depends_on" so that the stage gets created first, it complains about there being a cycle between the two.
So, it seems like:
aws_api_gateway_stage is only intended to add additional stages to a deployment, rather than creating a stage to use for the deployment
aws_api_gateway_deployment does not allow tags to be specified when it creates the stage.
Any ideas? What am I missing?
It seems that the stage_name field in the api_gateway_deployment should actually be optional. There is a PR open to fix the fact that its not at the moment. A workaround is at the moment to set stage_name to an empty string like this:
resource "aws_api_gateway_deployment" "PlayLambdaApiGatewayDeployment" {
depends_on = [
"aws_api_gateway_integration.PlayLambdaApiLambdaIntegration",
"aws_api_gateway_integration.PlayLambdaApiLambdaIntegrationRoot"
]
rest_api_id = "${aws_api_gateway_rest_api.PlayLambdaApiGateway.id}"
stage_name = ""
}
Like this there will be no additional stage created other than the one you specify in your aws_api_gateway_stage which you can set your tags for.