SNS topic subscription to AmazonIpSpaceChanged using terraform - amazon-web-services

I am trying to subscribe to the Aws AmazonIpSpaceChanged SNS topic using terraform. However, I keep getting the below error
SNS Topic subscription to AWS
resource "aws_sns_topic_subscription" "aws_ip_change_sns_subscription" {
topic_arn = "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged"
protocol = "lambda"
endpoint = "${aws_lambda_function.test_sg_lambda_function.arn}"
}
Error:
* module.test-lambda.aws_sns_topic_subscription.aws_ip_change_sns_subscription: 1 error(s) occurred:
* aws_sns_topic_subscription.aws_ip_change_sns_subscription: Error creating SNS topic: InvalidParameter: Invalid parameter: TopicArn
status code: 400, request id: 3daa2940-8d4b-5fd8-86e7-7b074a16ada9
I tried the same using aws cli and it failed the first time when I didn't include the option --region us-east-1. But once it is included, it was able to subscribe just fine.
Any ideas?

I know it's an old question, but there are no accepted answers - maybe this will help someone if you agree with it and mark it as accepted?
The SNS topic arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged is only available in the region us-east-1, so you need to use a provider within Terraform that is configured for that region.
You also need to give permissions to the SNS topic to invoke the Lambda function (not sure if you'd just left this off the question).
This also works if your lambda function is defined in a different region.
provider "aws" {
region = "{your target region}"
}
provider "aws" {
alias = "us_east_1"
region = "us-east-1"
}
resource "aws_lambda_function" "my_function" {
# This uses your default target region
:
:
}
resource "aws_lambda_permission" "lambda_permission" {
# This uses your default target region
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.my_function.function_name
principal = "sns.amazonaws.com"
source_arn = "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged"
}
resource "aws_sns_topic_subscription" "aws_ip_change_sns_subscription" {
# This needs to use the same region as the SNS topic
provider = aws.us_east_1
topic_arn = "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged"
protocol = "lambda"
endpoint = aws_lambda_function.my_function.arn
}

Your topic_arn is hardcoded to region us-east-1:
arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged
So when you have AWS_DEFAULT_REGION or similar configuration and point to another region, your code will fail.
That's the reason if you nominate the region, the code runs fine.
To avoid hardcodes, such as region, account id, you can do this:
data "aws_caller_identity" "current" {}
variable "region" {
type = "string"
default = "us-east-1"
}
resource "aws_sns_topic_subscription" "aws_ip_change_sns_subscription" {
topic_arn = "arn:aws:sns:${var.region}:${data.aws_caller_identity.current.account_id}:AmazonIpSpaceChanged"
protocol = "lambda"
endpoint = "${aws_lambda_function.test_sg_lambda_function.arn}"
}
With that, you should be fine and more flexible to run it in other region and other aws account as well.

Related

AWS Lambda is not sending error messages to AWS SQS DLQ

I'm trying to create via terraform, a lambda that triggered by Kinesis and her destination on failures will be AWS SQS.
I created and lambda and configured the source and destination
When I'm sending a message to Kinesis queue, the lambda is triggered but not sending messages to the DLQ.
What am I missing?
my labmda source mapping:
resource "aws_lambda_event_source_mapping" "csp_management_service_integration_stream_mapping" {
event_source_arn = local.kinesis_csp_management_service_integration_stream_arn
function_name = module.csp_management_service_integration_lambda.lambda_arn
batch_size = var.shared_kinesis_configuration.batch_size
bisect_batch_on_function_error = var.shared_kinesis_configuration.bisect_batch_on_function_error
starting_position = var.shared_kinesis_configuration.starting_position
maximum_retry_attempts = var.shared_kinesis_configuration.maximum_retry_attempts
maximum_record_age_in_seconds = var.shared_kinesis_configuration.maximum_record_age_in_seconds
function_response_types = var.shared_kinesis_configuration.function_response_types
destination_config {
on_failure {
destination_arn = local.shared_default_sqs_error_handling_dlq_arn
}
}
}
resource "aws_iam_policy" "shared_deadletter_sqs_queue_policy" {
name = "shared-deadletter-sqs-queue-policy"
path = "/"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"sqs:SendMessage",
]
Effect = "Allow"
Resource = [
local.shared_default_sqs_error_handling_dlq_arn
]
},
]
})
}
You should take a look on the following metric to see if you have permission error
I think you are facing some permission issue, try attaching a role to your lambda function with access to AWS SQS DLQ.
Is your DLQ encrypted by KMS? You will need top provide permissions to the KMS too in addition to SQS permissions
How is Lambda reporting failure?

Unable to find subscription for given ARN

I am testing my AWS terraform configuration with LocalStack. The final goal is to make a queue listen to my topic.
I am running Localstack with the following command:
docker run --rm -it -p 4566:4566 localstack/localstack
After running the command terraform destroy I get the error message:
aws_sns_topic_subscription.subscription: Destroying... [id=arn:aws:sns:us-east-1:000000000000:topic:a0d47652-3ae4-46df-9b63-3cb6e154cfcd]
╷
│ Error: error waiting for SNS topic subscription (arn:aws:sns:us-east-1:000000000000:topic:a0d47652-3ae4-46df-9b63-3cb6e154cfcd) deletion: InvalidParameter: Unable to find subscription for given ARN
│ status code: 400, request id: 2168e636
│
│
╵
I have run the code against the real AWS without a problem.
Here is the code for the terraform file
terraform {
required_version = ">= 0.12.26"
}
provider "aws" {
region = "us-east-1"
s3_force_path_style = true
skip_credentials_validation = true
skip_metadata_api_check = true
skip_requesting_account_id = true
endpoints {
sns = "http://localhost:4566"
sqs = "http://localhost:4566"
}
}
resource "aws_sqs_queue" "queue" {
name = "queue"
}
resource "aws_sns_topic" "topic" {
name = "topic"
}
resource "aws_sns_topic_subscription" "subscription" {
endpoint = aws_sqs_queue.queue.arn
protocol = "sqs"
topic_arn = aws_sns_topic.topic.arn
}
Sadly this is an issue with AWS, you have to create a ticket look here and https://stackoverflow.com/a/64568018/6085193
"When you delete a topic, subscriptions to the topic will not be "deleted" immediately, but become orphans. SNS will periodically clean these orphans, usually every 10 hours, but not guaranteed. If you create a new topic with the same topic name before these orphans are cleared up, the new topic will not inherit these orphans. So, no worry about them"
This has been fixed with issue:
https://github.com/localstack/localstack/issues/4022

Enable CloudWatch logs for AWS API Gateway using Terraform

I am using OpenAPI 3.0 spec to deploy an AWS API Gateway. I am not able to figure out how to enable cloud watch logs for the deployment.
Here is the terraform code:
data "template_file" "test_api_swagger" {
template = file(var.api_spec_path)
vars = {
//ommitted
}
}
resource "aws_api_gateway_rest_api" "test_api_gateway" {
name = "test_backend_api_gateway"
description = "API Gateway for some x"
body = data.template_file.test_api_swagger.rendered
endpoint_configuration {
types = ["REGIONAL"]
}
}
resource "aws_api_gateway_deployment" "test_lambda_gateway" {
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = var.env
}
I checked Amazon OpenAPI extensions and none seem to have this option. Only way I see is using api_gateway_method_settings which I cannot use in this case.
I think that it is not supported in terraform. I'm currently using terraform provisioner to run aws cli command after the deployment is created, like in the example below:
The example that I'm providing is to enable XRay tracing. You'll need to research the correct path and value to be used for CloudWatch logs. You can find more information in the docs.
resource "aws_api_gateway_deployment" "test_lambda_gateway" {
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = var.env
provisioner "local-exec" {
command = "aws apigateway update-stage --region ${data.aws_region.current.name} --rest-api-id ${aws_api_gateway_rest_api.test_api_gateway.id} --stage-name ${var.env} --patch-operations op=replace,path=/tracingEnabled,value=true"
}
}
You just need to make a reference to the aws data provider in your terraform template:
data "aws_region" "current" {}
Even though you're creating the gateway with OpenAPI import, you can still use api_gateway_method_settings to reference the stage, assuming you're using a stage as recommended. See AWS documentation. You would just indicate "*/*" on the method_path as per the example.
resource "aws_api_gateway_stage" "example" {
deployment_id = aws_api_gateway_deployment.test_lambda_gateway.id
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = "example"
}
resource "aws_api_gateway_method_settings" "all" {
rest_api_id = aws_api_gateway_rest_api.test_api_gateway.id
stage_name = aws_api_gateway_stage.example.stage_name
method_path = "*/*"
settings {
logging_level = "INFO"
}
}
This should set up the logging on the gateway for all requests with INFO level logging as if you had done it in the console on the stage.

Creating the topic rule does not create the trigger on the lambda

This issue looks very much like a bug but I believe there must be something wrong in my terraform file because I can't find anybody on the web having the same problem.
Here is the part of my terraform file that creates a lambda and a topic rule for it:
resource "aws_lambda_function" "rds_persist" {
filename = "${local.rds_persist_file_path}"
function_name = "RdsPersist-${var.env}"
role = "${aws_iam_role.lambda_role.arn}"
handler = "package.handler"
source_code_hash = "${local.rds_persist_package_hash}"
runtime = "nodejs8.10"
memory_size = 128
timeout = 10
vpc_config = {
subnet_ids = ["${var.private_subnet_ids}"]
security_group_ids = ["${aws_security_group.all_vpc_access.id}"]
}
environment {
variables = {
DB = "${var.database_url}"
IOT_DEVICE_ARN = "${var.iot_device_v1_sns_arn}"
}
}
}
resource "aws_iot_topic_rule" "rds_push" {
name = "${var.env}_RdsPush"
description = "Pushes events to a persistence lambda (rds store)"
enabled = true
sql = "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), '${var.env}-')"
sql_version = "2016-03-23"
lambda {
function_arn = "${aws_lambda_function.rds_persist.arn}"
}
}
Here is the result in AWS Console:
If I delete and re-add the rule in the console, then the trigger appears on the lambda.
It might be the case your lambda function that your topic uses is being created before the function.
I tested this also by tainting the topic rule alone so it got recreated (see the logs below). Unfortunately it didn't solve the issue.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
-/+ module.lambda.aws_iot_topic_rule.rds_push (tainted) (new resource required)
id: "dev_RdsPush" => <computed> (forces new resource)
arn: "arn:aws:iot:eu-west-1:827689093226:rule/dev_RdsPush" => <computed>
description: "Pushes events to a persistence lambda (rds store)" => "Pushes events to a persistence lambda (rds store)"
enabled: "true" => "true"
lambda.#: "1" => "1"
lambda.1860721139.function_arn: "arn:aws:lambda:eu-west-1:827689093226:function:RdsPersist-dev" => "arn:aws:lambda:eu-west-1:827689093226:function:RdsPersist-dev"
name: "dev_RdsPush" => "dev_RdsPush"
sql: "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), 'dev-')" => "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), 'dev-')"
sql_version: "2016-03-23" => "2016-03-23"
Plan: 1 to add, 0 to change, 1 to destroy.
Update: I just found a very similar issue at a different place:
There's supposed to be an SNS subscription between this another lambda and a SNS. Here is the relevant code in the terraform:
resource "aws_sns_topic_subscription" "conference_call" {
topic_arn = "${module.sns.conference_call_arn}"
protocol = "lambda"
endpoint = "${module.lambda.messaging_arn}"
}
(Obviously I checked the resources and they are correct)
In the console I don't see the trigger in the lambda but I do see the subscription in SNS:
Update: exact same issue when creating the resources using AWS CLI
# For the first issue
$ aws iot create-topic-rule --rule-name dev_RdsPush --topic-rule-payload '{"sql":"SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM \'#\' WHERE startswith(clientid(), \'dev-\')","actions":[{"lambda":{"functionArn":"arn:aws:lambda:eu-west-1:xxxxxxxxx:function:RdsPersist-dev"}}]}'
# For the second issue
$ aws sns subscribe --topic-arn arn:aws:sns:eu-west-1:xxxxxxxx:conference-call-v1-dev --protocol lambda --notification-endpoint arn:aws:lambda:eu-west-1:xxxxxxxxx:function:Messaging-dev
Solution:
Add these:
IoT:
resource "aws_lambda_permission" "conference_call_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.conference_call.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${var.conference_call_sns_arn}"
}
SNS:
resource "aws_lambda_permission" "messaging_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.messaging.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${var.conference_call_sns_arn}"
}
You need to add a lambda permission to allow IoT to invoke lambda. The Lambda console uses the permissions of the function to show what can invoke it.
https://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.html#lambda-rule.
It might be the case your lambda function that your topic uses is being created before the function. Try adding depends_on = ["aws_lambda_function.rds_persist"] on your aws_iot_topic_rule and see how it goes.
In my case this was the missing permission:
resource "aws_lambda_permission" "rds_topic_rule_permission" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.rds_persist.function_name
principal = "iot.amazonaws.com"
source_arn = aws_iot_topic_rule.rds_push.arn
}
See: https://github.com/hashicorp/terraform-provider-aws/issues/24196#issuecomment-1103979956

AWS SNS ought to trigger my lambda, but does not

I have an AWS lambda function that I created via apex. I've also created a SNS topic and a subscription through terraform.
My topic is: arn:aws:sns:ap-southeast-1:178284945954:fetch_realm_auctions
I have a subscription: arn:aws:sns:ap-southeast-1:178284945954:fetch_realm_auctions:2da1d182-946d-4afd-91cb-1ed3453c5d86 with a lambda type and the endpoint is: arn:aws:lambda:ap-southeast-1:178284945954:function:wowauctions_get_auction_data
I've confirmed this is the correct function ARN. Everything seems wired up correctly:
I trigger SNS manually:
aws sns publish
--topic-arn arn:aws:sns:ap-southeast-1:178284945954:fetch_realm_auctions
--message '{"endpoint": "https://us.api.battle.net", "realm": "spinebreaker"}'
It returns the message ID but no invocation happens. Why?
I added an inline policy to allow the lambda to be invoked:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1474873816000",
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction"
],
"Resource": [
"arn:aws:lambda:ap-southeast-1:178284945954:function:wowauctions_get_auction_data"
]
}
]
}
And it's now working.
The SNS topic needs to have the permission to invoke the Lambda.
Here is an example how you can express that in Terraform:
# Assumption: both SNS topic and Lambda are deployed in the same region
# resource "aws_sns_topic" "instance" { ... }
# resource "aws_lambda_function" "instance" {... }
# Step 1: Allow the SNS topic to invoke the Lambda
resource "aws_lambda_permission" "allow_invocation_from_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.instance.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${aws_sns_topic.instance.arn}"
}
# Step 2: Subscribe the Lambda to the SNS topic
resource "aws_sns_topic_subscription" "instance" {
topic_arn = "${aws_sns_topic.instance.arn}"
protocol = "lambda"
endpoint = "${aws_lambda_function.instance.arn}"
}
Some general tips for troubleshooting this problem (a Lambda not being fired):
Does my message arrive at the Lambda? -- Subscribe your email address to the SNS topic. If you get emails, you will know when messages arrive at the topic.
Is the Lambda subscribed to the topic? -- Check in the AWS console (under SNS -> Topic) whether the subscription is correct (the endpoint must exactly match the ARN of the Lambda)
Once you confirmed these basic checks and you still see no invocations, it has to be a permission error. When you open the Lambda in the AWS console, you should see SNS listed as a trigger:
For comparison, if the permission is missing, you will not see SNS:
If you are not using an automated deployment (e.g., with CloudFormation or Terraform), you can also manually add the missing permission:
Choose SNS under Add triggers (you will need to scroll down in the list to see it)
In Configure triggers, select the SNS topic
Click Add and save the Lambda
For me the problem was that I specified SourceAccount parameter inside AWS::Lambda::Permission in my cloudformation template and documentation states the following:
Do not use the --source-account parameter to add a source account to the Lambda policy when adding the policy. Source account is not supported for Amazon SNS event sources and will result in access being denied. This has no security impact as the source account is included in the source ARN.
As soon as I removed SourceAccount, everything worked fine.
As Robo mentioned in the comments, adding a Principal based permission is the simplest way of doing this:
"FooFunctionPermission" : {
"Type" : "AWS::Lambda::Permission",
"Properties" : {
"Action" : "lambda:InvokeFunction",
"FunctionName" : { "Ref" : "FooFunction" },
"Principal" : "sns.amazonaws.com"
}
}
Had the same issue:
1) Created and deployed simple lambda
2) Created aws sns topic manually from java sdk
3) Created sns subscription from java sdk (subscription between sns topic and
lambda)
Then I had a problem, when pushed some message to the topic from the console - it was not intercepted by lambda. And more, sns trigger was not even registered in the lambda.
So I fixed this simply by using this command:
https://docs.aws.amazon.com/cli/latest/reference/lambda/add-permission.html
After running aws lambda add-permission ......, everything were picked up and working fine.
This post helped me to get farther, but there is a missing piece. Terraform will create the wrong subscription. You must drop $LATEST
resource "aws_sns_topic" "cloudwatch_notifications" {
name = "aws-${var.service_name}-${var.stage}-alarm"
}
data "aws_lambda_function" "cloudwatch_lambda" {
function_name = "sls-${var.service_name}-${var.stage}-cloudwatch-alarms"
}
resource "aws_lambda_permission" "with_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${replace(data.aws_lambda_function.cloudwatch_lambda.arn, ":$LATEST", "")}"
principal = "sns.amazonaws.com"
source_arn = "${aws_sns_topic.cloudwatch_notifications.arn}"
}
resource "aws_sns_topic_subscription" "cloudwatch_subscription" {
topic_arn = "${aws_sns_topic.cloudwatch_notifications.arn}"
protocol = "lambda"
endpoint = "${replace(data.aws_lambda_function.cloudwatch_lambda.arn, ":$LATEST", "")}"
}
This is a specific answer to this question - I have removed my other answer elsewhere !
For Terraform users, see also here:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission
which shows use of the 'aws_lambda_permission' resource; SNS is covered in one of the examples, copied here:
resource "aws_lambda_permission" "with_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.func.function_name
principal = "sns.amazonaws.com"
source_arn = aws_sns_topic.default.arn
}