I'm trying to create via terraform, a lambda that triggered by Kinesis and her destination on failures will be AWS SQS.
I created and lambda and configured the source and destination
When I'm sending a message to Kinesis queue, the lambda is triggered but not sending messages to the DLQ.
What am I missing?
my labmda source mapping:
resource "aws_lambda_event_source_mapping" "csp_management_service_integration_stream_mapping" {
event_source_arn = local.kinesis_csp_management_service_integration_stream_arn
function_name = module.csp_management_service_integration_lambda.lambda_arn
batch_size = var.shared_kinesis_configuration.batch_size
bisect_batch_on_function_error = var.shared_kinesis_configuration.bisect_batch_on_function_error
starting_position = var.shared_kinesis_configuration.starting_position
maximum_retry_attempts = var.shared_kinesis_configuration.maximum_retry_attempts
maximum_record_age_in_seconds = var.shared_kinesis_configuration.maximum_record_age_in_seconds
function_response_types = var.shared_kinesis_configuration.function_response_types
destination_config {
on_failure {
destination_arn = local.shared_default_sqs_error_handling_dlq_arn
}
}
}
resource "aws_iam_policy" "shared_deadletter_sqs_queue_policy" {
name = "shared-deadletter-sqs-queue-policy"
path = "/"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"sqs:SendMessage",
]
Effect = "Allow"
Resource = [
local.shared_default_sqs_error_handling_dlq_arn
]
},
]
})
}
You should take a look on the following metric to see if you have permission error
I think you are facing some permission issue, try attaching a role to your lambda function with access to AWS SQS DLQ.
Is your DLQ encrypted by KMS? You will need top provide permissions to the KMS too in addition to SQS permissions
How is Lambda reporting failure?
Related
I have a SQS Terraform module in which I defined the queue name as below
main_queue_name = "app-sqs-env-${var.env_name}"
by defining the env_name in a separate file and I am able to create a queue with the desired name.
Now I want to create an SNS topic and want the queue to be subscribed to this topic.
when I create the SNS topic using sns_topic_name = "app-sns-env-${var.env_name}" I an able to create the topic as expected
How do I define the sqs_endpoint in the SNS module, I want to use ${var.env_name} in this endpoint definition as we pass different names for different environments.
In order to be able to subscribe an SQS queue to an SNS topic we have to do the following:
# Create some locals for SQS and SNS names
locals {
sqs-name = "app-sqs-env-${var.env-name}"
sns-name = "app-sns-env-${var.env-name}"
}
# Inject caller ID for being able to use the account ID
data "aws_caller_identity" "current" {}
# Create a topic policy. This will allow for the SQS queue to be able to subscribe to the topic
data "aws_iam_policy_document" "sns-topic-policy" {
statement {
actions = [
"SNS:Subscribe",
"SNS:Receive",
]
condition {
test = "StringLike"
variable = "SNS:Endpoint"
# In order to avoid circular dependencies, we must create the ARN ourselves
values = [
"arn:aws:sqs:${var.region}:${data.aws_caller_identity.current.account_id}:${local.sqs-name}",
]
}
effect = "Allow"
principals {
type = "AWS"
identifiers = ["*"]
}
resources = [
"arn:aws:sns:${var.region}:${data.aws_caller_identity.current.account_id}:${local.sns-name}"
]
sid = "sid-101"
}
}
# Create a queue policy. This allows for the SNS topic to be able to publish messages to the SQS queue
data "aws_iam_policy_document" "sqs-queue-policy" {
policy_id = "arn:aws:sqs:${var.region}:${data.aws_caller_identity.current.account_id}:${local.sqs-name}/SQSDefaultPolicy"
statement {
sid = "example-sns-topic"
effect = "Allow"
principals {
type = "AWS"
identifiers = ["*"]
}
actions = [
"SQS:SendMessage",
]
resources = [
"arn:aws:sqs:${var.region}:${data.aws_caller_identity.current.account_id}:${local.sqs-name}"
]
condition {
test = "ArnEquals"
variable = "aws:SourceArn"
values = [
"arn:aws:sns:${var.region}:${data.aws_caller_identity.current.account_id}:${local.sns-name}"
]
}
}
}
# Create the SNS topic and assign the topic policy to it
resource "aws_sns_topic" "sns-topic" {
name = local.sns-name
display_name = local.sns-name
policy = data.aws_iam_policy_document.sns-topic-policy.json
}
# Create the SQS queue and assign the queue policy to it
resource "aws_sqs_queue" "sqs-queue" {
name = local.sqs-name
policy = data.aws_iam_policy_document.sqs-queue-policy.json
}
# Subscribe the SQS queue to the SNS topic
resource "aws_sns_topic_subscription" "sns-topic" {
topic_arn = aws_sns_topic.sns-topic.arn
protocol = "sqs"
endpoint = aws_sqs_queue.sqs-queue.arn
}
I hope the code and the comments above make sense. There is an example on the Terraform documentation for aws_sns_topic_subscription which is way more complex, but also usable.
I had quite a hard time setting up an automization with Beanstalk and Codepipeline...
I finally got it running, the main issue was the S3 Cloudwatch event to trigger the start of the Codepipeline. I missed the Cloudtrail part which is necessary and I couldn't find that in any documentation.
So the current Setup is:
S3 file gets uploaded -> a CloudWatch Event triggers the Codepipeline -> Codepipeline deploys to ElasticBeanstalk env.
As I said to get the CloudWatch Event trigger you need a Cloudtrail trail like:
resource "aws_cloudtrail" "example" {
# ... other configuration ...
name = "codepipeline-source-trail" #"codepipeline-${var.project_name}-trail"
is_multi_region_trail = true
s3_bucket_name = "codepipeline-cloudtrail-placeholder-bucket-eu-west-1"
event_selector {
read_write_type = "WriteOnly"
include_management_events = true
data_resource {
type = "AWS::S3::Object"
values = ["${data.aws_s3_bucket.bamboo-deploy-bucket.arn}/${var.project_name}/file.zip"]
}
}
}
But this is only to create a new trail. The problem is that AWS only allows 5 trails max. On the AWS console you can add multiple data events to one trail, but I couldn't manage to do this in terraform. I tried to use the same name, but this just raises an error
"Error creating CloudTrail: TrailAlreadyExistsException: Trail codepipeline-source-trail already exists for customer: XXXX"
I tried my best to explain my problem. Not sure if it is understandable.
In a nutshell: I want to add a data events:S3 in an existing cloudtrail trail with terraform.
Thx for help,
Daniel
As I said to get the CloudWatch Event trigger you need a Cloudtrail trail like:
You do not need multiple CloudTrail to invoke a CloudWatch Event. You can create service-specific rules as well.
Create a CloudWatch Events rule for an Amazon S3 source (console)
From CloudWatch event rule to invoke CodePipeline as a target. Let's say you created this event rule
{
"source": [
"aws.s3"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"s3.amazonaws.com"
],
"eventName": [
"PutObject"
]
}
}
You add CodePipeline as a target for this rule and eventually, Codepipeline deploys to ElasticBeanstalk env.
Have you tried to add multiple data_resources to your current trail instead of adding a new trail with the same name:
resource "aws_cloudtrail" "example" {
# ... other configuration ...
name = "codepipeline-source-trail" #"codepipeline-${var.project_name}-trail"
is_multi_region_trail = true
s3_bucket_name = "codepipeline-cloudtrail-placeholder-bucket-eu-west-1"
event_selector {
read_write_type = "WriteOnly"
include_management_events = true
data_resource {
type = "AWS::S3::Object"
values = ["${data.aws_s3_bucket.bamboo-deploy-bucket.arn}/${var.project_A}/file.zip"]
}
data_resource {
type = "AWS::S3::Object"
values = ["${data.aws_s3_bucket.bamboo-deploy-bucket.arn}/${var.project_B}/fileB.zip"]
}
}
}
You should be able to add up to 250 data resources (across all event selectors in a trail), and up to 5 event selectors to your current trail (CloudTrail quota limits)
This issue looks very much like a bug but I believe there must be something wrong in my terraform file because I can't find anybody on the web having the same problem.
Here is the part of my terraform file that creates a lambda and a topic rule for it:
resource "aws_lambda_function" "rds_persist" {
filename = "${local.rds_persist_file_path}"
function_name = "RdsPersist-${var.env}"
role = "${aws_iam_role.lambda_role.arn}"
handler = "package.handler"
source_code_hash = "${local.rds_persist_package_hash}"
runtime = "nodejs8.10"
memory_size = 128
timeout = 10
vpc_config = {
subnet_ids = ["${var.private_subnet_ids}"]
security_group_ids = ["${aws_security_group.all_vpc_access.id}"]
}
environment {
variables = {
DB = "${var.database_url}"
IOT_DEVICE_ARN = "${var.iot_device_v1_sns_arn}"
}
}
}
resource "aws_iot_topic_rule" "rds_push" {
name = "${var.env}_RdsPush"
description = "Pushes events to a persistence lambda (rds store)"
enabled = true
sql = "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), '${var.env}-')"
sql_version = "2016-03-23"
lambda {
function_arn = "${aws_lambda_function.rds_persist.arn}"
}
}
Here is the result in AWS Console:
If I delete and re-add the rule in the console, then the trigger appears on the lambda.
It might be the case your lambda function that your topic uses is being created before the function.
I tested this also by tainting the topic rule alone so it got recreated (see the logs below). Unfortunately it didn't solve the issue.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
-/+ module.lambda.aws_iot_topic_rule.rds_push (tainted) (new resource required)
id: "dev_RdsPush" => <computed> (forces new resource)
arn: "arn:aws:iot:eu-west-1:827689093226:rule/dev_RdsPush" => <computed>
description: "Pushes events to a persistence lambda (rds store)" => "Pushes events to a persistence lambda (rds store)"
enabled: "true" => "true"
lambda.#: "1" => "1"
lambda.1860721139.function_arn: "arn:aws:lambda:eu-west-1:827689093226:function:RdsPersist-dev" => "arn:aws:lambda:eu-west-1:827689093226:function:RdsPersist-dev"
name: "dev_RdsPush" => "dev_RdsPush"
sql: "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), 'dev-')" => "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), 'dev-')"
sql_version: "2016-03-23" => "2016-03-23"
Plan: 1 to add, 0 to change, 1 to destroy.
Update: I just found a very similar issue at a different place:
There's supposed to be an SNS subscription between this another lambda and a SNS. Here is the relevant code in the terraform:
resource "aws_sns_topic_subscription" "conference_call" {
topic_arn = "${module.sns.conference_call_arn}"
protocol = "lambda"
endpoint = "${module.lambda.messaging_arn}"
}
(Obviously I checked the resources and they are correct)
In the console I don't see the trigger in the lambda but I do see the subscription in SNS:
Update: exact same issue when creating the resources using AWS CLI
# For the first issue
$ aws iot create-topic-rule --rule-name dev_RdsPush --topic-rule-payload '{"sql":"SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM \'#\' WHERE startswith(clientid(), \'dev-\')","actions":[{"lambda":{"functionArn":"arn:aws:lambda:eu-west-1:xxxxxxxxx:function:RdsPersist-dev"}}]}'
# For the second issue
$ aws sns subscribe --topic-arn arn:aws:sns:eu-west-1:xxxxxxxx:conference-call-v1-dev --protocol lambda --notification-endpoint arn:aws:lambda:eu-west-1:xxxxxxxxx:function:Messaging-dev
Solution:
Add these:
IoT:
resource "aws_lambda_permission" "conference_call_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.conference_call.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${var.conference_call_sns_arn}"
}
SNS:
resource "aws_lambda_permission" "messaging_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.messaging.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${var.conference_call_sns_arn}"
}
You need to add a lambda permission to allow IoT to invoke lambda. The Lambda console uses the permissions of the function to show what can invoke it.
https://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.html#lambda-rule.
It might be the case your lambda function that your topic uses is being created before the function. Try adding depends_on = ["aws_lambda_function.rds_persist"] on your aws_iot_topic_rule and see how it goes.
In my case this was the missing permission:
resource "aws_lambda_permission" "rds_topic_rule_permission" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.rds_persist.function_name
principal = "iot.amazonaws.com"
source_arn = aws_iot_topic_rule.rds_push.arn
}
See: https://github.com/hashicorp/terraform-provider-aws/issues/24196#issuecomment-1103979956
I am trying to subscribe to the Aws AmazonIpSpaceChanged SNS topic using terraform. However, I keep getting the below error
SNS Topic subscription to AWS
resource "aws_sns_topic_subscription" "aws_ip_change_sns_subscription" {
topic_arn = "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged"
protocol = "lambda"
endpoint = "${aws_lambda_function.test_sg_lambda_function.arn}"
}
Error:
* module.test-lambda.aws_sns_topic_subscription.aws_ip_change_sns_subscription: 1 error(s) occurred:
* aws_sns_topic_subscription.aws_ip_change_sns_subscription: Error creating SNS topic: InvalidParameter: Invalid parameter: TopicArn
status code: 400, request id: 3daa2940-8d4b-5fd8-86e7-7b074a16ada9
I tried the same using aws cli and it failed the first time when I didn't include the option --region us-east-1. But once it is included, it was able to subscribe just fine.
Any ideas?
I know it's an old question, but there are no accepted answers - maybe this will help someone if you agree with it and mark it as accepted?
The SNS topic arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged is only available in the region us-east-1, so you need to use a provider within Terraform that is configured for that region.
You also need to give permissions to the SNS topic to invoke the Lambda function (not sure if you'd just left this off the question).
This also works if your lambda function is defined in a different region.
provider "aws" {
region = "{your target region}"
}
provider "aws" {
alias = "us_east_1"
region = "us-east-1"
}
resource "aws_lambda_function" "my_function" {
# This uses your default target region
:
:
}
resource "aws_lambda_permission" "lambda_permission" {
# This uses your default target region
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.my_function.function_name
principal = "sns.amazonaws.com"
source_arn = "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged"
}
resource "aws_sns_topic_subscription" "aws_ip_change_sns_subscription" {
# This needs to use the same region as the SNS topic
provider = aws.us_east_1
topic_arn = "arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged"
protocol = "lambda"
endpoint = aws_lambda_function.my_function.arn
}
Your topic_arn is hardcoded to region us-east-1:
arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged
So when you have AWS_DEFAULT_REGION or similar configuration and point to another region, your code will fail.
That's the reason if you nominate the region, the code runs fine.
To avoid hardcodes, such as region, account id, you can do this:
data "aws_caller_identity" "current" {}
variable "region" {
type = "string"
default = "us-east-1"
}
resource "aws_sns_topic_subscription" "aws_ip_change_sns_subscription" {
topic_arn = "arn:aws:sns:${var.region}:${data.aws_caller_identity.current.account_id}:AmazonIpSpaceChanged"
protocol = "lambda"
endpoint = "${aws_lambda_function.test_sg_lambda_function.arn}"
}
With that, you should be fine and more flexible to run it in other region and other aws account as well.
I have an AWS lambda function that I created via apex. I've also created a SNS topic and a subscription through terraform.
My topic is: arn:aws:sns:ap-southeast-1:178284945954:fetch_realm_auctions
I have a subscription: arn:aws:sns:ap-southeast-1:178284945954:fetch_realm_auctions:2da1d182-946d-4afd-91cb-1ed3453c5d86 with a lambda type and the endpoint is: arn:aws:lambda:ap-southeast-1:178284945954:function:wowauctions_get_auction_data
I've confirmed this is the correct function ARN. Everything seems wired up correctly:
I trigger SNS manually:
aws sns publish
--topic-arn arn:aws:sns:ap-southeast-1:178284945954:fetch_realm_auctions
--message '{"endpoint": "https://us.api.battle.net", "realm": "spinebreaker"}'
It returns the message ID but no invocation happens. Why?
I added an inline policy to allow the lambda to be invoked:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1474873816000",
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction"
],
"Resource": [
"arn:aws:lambda:ap-southeast-1:178284945954:function:wowauctions_get_auction_data"
]
}
]
}
And it's now working.
The SNS topic needs to have the permission to invoke the Lambda.
Here is an example how you can express that in Terraform:
# Assumption: both SNS topic and Lambda are deployed in the same region
# resource "aws_sns_topic" "instance" { ... }
# resource "aws_lambda_function" "instance" {... }
# Step 1: Allow the SNS topic to invoke the Lambda
resource "aws_lambda_permission" "allow_invocation_from_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.instance.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${aws_sns_topic.instance.arn}"
}
# Step 2: Subscribe the Lambda to the SNS topic
resource "aws_sns_topic_subscription" "instance" {
topic_arn = "${aws_sns_topic.instance.arn}"
protocol = "lambda"
endpoint = "${aws_lambda_function.instance.arn}"
}
Some general tips for troubleshooting this problem (a Lambda not being fired):
Does my message arrive at the Lambda? -- Subscribe your email address to the SNS topic. If you get emails, you will know when messages arrive at the topic.
Is the Lambda subscribed to the topic? -- Check in the AWS console (under SNS -> Topic) whether the subscription is correct (the endpoint must exactly match the ARN of the Lambda)
Once you confirmed these basic checks and you still see no invocations, it has to be a permission error. When you open the Lambda in the AWS console, you should see SNS listed as a trigger:
For comparison, if the permission is missing, you will not see SNS:
If you are not using an automated deployment (e.g., with CloudFormation or Terraform), you can also manually add the missing permission:
Choose SNS under Add triggers (you will need to scroll down in the list to see it)
In Configure triggers, select the SNS topic
Click Add and save the Lambda
For me the problem was that I specified SourceAccount parameter inside AWS::Lambda::Permission in my cloudformation template and documentation states the following:
Do not use the --source-account parameter to add a source account to the Lambda policy when adding the policy. Source account is not supported for Amazon SNS event sources and will result in access being denied. This has no security impact as the source account is included in the source ARN.
As soon as I removed SourceAccount, everything worked fine.
As Robo mentioned in the comments, adding a Principal based permission is the simplest way of doing this:
"FooFunctionPermission" : {
"Type" : "AWS::Lambda::Permission",
"Properties" : {
"Action" : "lambda:InvokeFunction",
"FunctionName" : { "Ref" : "FooFunction" },
"Principal" : "sns.amazonaws.com"
}
}
Had the same issue:
1) Created and deployed simple lambda
2) Created aws sns topic manually from java sdk
3) Created sns subscription from java sdk (subscription between sns topic and
lambda)
Then I had a problem, when pushed some message to the topic from the console - it was not intercepted by lambda. And more, sns trigger was not even registered in the lambda.
So I fixed this simply by using this command:
https://docs.aws.amazon.com/cli/latest/reference/lambda/add-permission.html
After running aws lambda add-permission ......, everything were picked up and working fine.
This post helped me to get farther, but there is a missing piece. Terraform will create the wrong subscription. You must drop $LATEST
resource "aws_sns_topic" "cloudwatch_notifications" {
name = "aws-${var.service_name}-${var.stage}-alarm"
}
data "aws_lambda_function" "cloudwatch_lambda" {
function_name = "sls-${var.service_name}-${var.stage}-cloudwatch-alarms"
}
resource "aws_lambda_permission" "with_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${replace(data.aws_lambda_function.cloudwatch_lambda.arn, ":$LATEST", "")}"
principal = "sns.amazonaws.com"
source_arn = "${aws_sns_topic.cloudwatch_notifications.arn}"
}
resource "aws_sns_topic_subscription" "cloudwatch_subscription" {
topic_arn = "${aws_sns_topic.cloudwatch_notifications.arn}"
protocol = "lambda"
endpoint = "${replace(data.aws_lambda_function.cloudwatch_lambda.arn, ":$LATEST", "")}"
}
This is a specific answer to this question - I have removed my other answer elsewhere !
For Terraform users, see also here:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission
which shows use of the 'aws_lambda_permission' resource; SNS is covered in one of the examples, copied here:
resource "aws_lambda_permission" "with_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.func.function_name
principal = "sns.amazonaws.com"
source_arn = aws_sns_topic.default.arn
}