I have setup the following in Terraform. So two event rules, start_event at 8am and stop_event at 6pm.
# Create cloudwatch event rules
resource "aws_cloudwatch_event_rule" "stop_ec2_event_rule" {
name = "stop-ec2-event-rule"
description = "Stop EC2 instance at a specified time each day"
schedule_expression = var.cloudwatch_schedule_stop
}
resource "aws_cloudwatch_event_rule" "start_ec2_event_rule" {
name = "start-ec2-event-rule"
description = "Start EC2 instance at a specified time each day"
schedule_expression = var.cloudwatch_schedule_start
}
Each event passes an action to the lambda
resource "aws_cloudwatch_event_target" "stop_ec2_event_rule_target" {
rule = aws_cloudwatch_event_rule.stop_ec2_event_rule.name
target_id = "TriggerLambdaFunction"
arn = aws_lambda_function.lambda_rscheduler.arn
input = "{\"environment\":\"${var.environment}\", \"action\":\"stop\"}"
}
resource "aws_cloudwatch_event_target" "start_ec2_event_rule_target" {
rule = aws_cloudwatch_event_rule.start_ec2_event_rule.name
target_id = "TriggerLambdaFunction"
arn = aws_lambda_function.lambda_rscheduler.arn
input = "{\"environment\":\"${var.environment}\", \"action\":\"start\"}"
}
This works
resource "aws_lambda_permission" "allow_cloudwatch" {
statement_id = "AllowExecutionFromCloudWatch"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.lambda_rscheduler.function_name
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.stop_ec2_event_rule.arn
This issue I'm facing is that I cannot get Terraform to associate the start_event with the lambda function. I go into the AWS console and I can manually add the CloudWatch start_event trigger to the lambda function.
If I have the start_event resources
resource "aws_lambda_permission" "allow_cloudwatch" {
statement_id = "AllowExecutionFromCloudWatch"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.lambda_rscheduler.function_name
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.start_ec2_event_rule.arn
It will complain that the statement_id is duplicated.
I needed something like the terraform aws_lambda_event_source_mapping but that only allows Lambda functions to get events from Kinesis, DynamoDB and SQS; and not a CloudWatch event.
How can I tell terraform to associate multiple cloudwatch events to the same lambda function; when I can manually do it from the AWS console?
statement_id is not compulsory, so you can safely omit it from your aws_lambda_permission and terraform will unique generate id for you automatically. You can also use count or for_each to save you some typing for aws_lambda_permission.
For example, using for_each you could define aws_lambda_permission to be:
resource "aws_lambda_permission" "allow_cloudwatch" {
for_each = {for idx, v in [
aws_cloudwatch_event_rule.stop_ec2_event_rule,
aws_cloudwatch_event_rule.start_ec2_event_rule
]: idx => v}
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.lambda_rscheduler.function_name
principal = "events.amazonaws.com"
source_arn = each.value.arn
}
Analogical versions could be written for aws_cloudwatch_event_rule and aws_cloudwatch_event_target so that your code is based on for_each or count without the copy-and-paste repetitions.
Related
I am deploying a REST API Gateway using Terraform. Couple of endpoints are accessing Lambda function to return response. Whenever I deploy api-gw using terraform, the Lambda permission doesn't seem to refresh and I have to manually open the api-gw portal in AWS console and again add that lambda function post which it prompts me to allow invoke action. How can I refresh the permission without having to do these manual steps ? I am using below snippet for api-gw deployment and lambda permissions:
resource "aws_api_gateway_deployment" "deploy" {
rest_api_id = aws_api_gateway_rest_api.apigw.id
stage_name = ""
variables = {
deployed_at = timestamp()
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_lambda_permission" "customers_lambda_permission" {
statement_id = "AllowDemoAPIInvokeProjectGet"
action = "lambda:InvokeFunction"
function_name = local.lambda_name
principal = "apigateway.amazonaws.com"
source_arn = "${aws_api_gateway_rest_api.apigw.execution_arn}/*/GET/api/customers"
}
Your aws_api_gateway_deployment resource should depend on the aws_api_gateway_integration so that the lambda integration is created before deployment.
resource "aws_api_gateway_deployment" "deploy" {
...
depends_on = [
aws_api_gateway_integration.example1,
aws_api_gateway_integration.example2
]
}
or use triggers attribute:
resource "aws_api_gateway_deployment" "deploy" {
...
triggers = {
redeployment = sha1(jsonencode([
aws_api_gateway_resource.example1.id,
aws_api_gateway_method.example1.id,
aws_api_gateway_integration.example1.id,
]))
}
I have defined the creation of a StepFunction state machine in Terraform, now I want to set a timer to trigger the state machine everyday, I think probably using cloudwatch event rules is a good choice, I know how to set event rule to trigger a Lambda:
resource "aws_cloudwatch_event_rule" "lambda_event_rule" {
name = xxx
schedule_expression = xxx
description = xxx
}
resource "aws_cloudwatch_event_target" "lambda_event_target" {
target_id = xxx
rule = aws_cloudwatch_event_rule.lambda_event_rule.name
arn = xxx
}
#I must setup the right permissions using 'aws_lambda_permission'
#see: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_target
resource "aws_lambda_permission" "lambda_event_permission" {
statement_id = xxx
action = "lambda:InvokeFunction"
function_name = xxx
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.lambda_event_rule.name
}
but how can I setup the permission part for triggerring a state machine? I couldn't find any examples about it, am I missing anything? Is it because we don't need a permission config for state machine? Can someone help please?
Below is what I got to use cloudwatch event rules to trigger state machine so far:
resource "aws_cloudwatch_event_rule" "step_function_event_rule" {
name = xxx
schedule_expression = xxx
description = xxx
}
resource "aws_cloudwatch_event_target" "step_function_event_target" {
target_id = xxx
rule = aws_cloudwatch_event_rule.step_function_event_rule.name
arn = xxx
}
?????What else should I add here?
PS: I found someone else was asking about a similar question here, but no answers yet.
The
resource "aws_lambda_permission" "lambda_event_permission" {
statement_id = xxx
action = "lambda:InvokeFunction"
function_name = xxx
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.lambda_event_rule.name
}
part is not needed at all in your case, only needed as stated "In order to be able to have your AWS Lambda function or SNS topic invoked by an EventBridge rule".
As blr stated in his answer, you need to add the role_arn in the aws_cloudwatch_event_target, set up a role with assume_role_policy which grants access to states.amazonaws.com and events.amazonaws.com, and attach to this role an extra policy as follows:
data "aws_iam_policy_document" "CW2SF_allowexec" {
statement {
actions = [
"sts:AssumeRole"
]
principals {
type = "Service"
identifiers = [
"states.amazonaws.com",
"events.amazonaws.com"
]
}
}
}
resource "aws_iam_role" "CW2SF_allowexec" {
name = "AWS_Events_Invoke-StepFunc"
assume_role_policy = data.aws_iam_policy_document.CW2SF_allowexec.json
}
resource "aws_iam_role_policy" "state-execution" {
name = "CW2SF_allowexec"
role = aws_iam_role.CW2SF_allowexec.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"states:StartExecution"
],
"Resource": [
"arn:aws:states:${var.region}:${data.aws_caller_identity.current.account_id}:stateMachine:data-pipeline-incremental"
]
}
]
}
EOF
}
You need to establish the trust between CloudWatch and StepFunctions with the AssumeRole, and then attach an inline or managed policy to the role that specifically allows this role to StartExecution of the state machine.
I'm not well versed with terraform but it seems to follow a similar pattern to the official documentation. For targets; https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_PutTargets.html >> See section "Adds a Step Functions state machine as a target"
{
"Rule": "testrule",
"Targets": [
{
"RoleArn": "arn:aws:iam::123456789012:role/MyRoleToAccessStepFunctions"
"Arn":"arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorld"
}
]
}
This tells me that you need to pass the role and arn. So taking your example, here's the thing you probably need to fill
resource "aws_cloudwatch_event_rule" "step_function_event_rule" {
name = <something unique>
schedule_expression = <syntax described in https://docs.aws.amazon.com/eventbridge/latest/userguide/scheduled-events.html>
description = <something descriptive>
}
resource "aws_cloudwatch_event_target" "step_function_event_target" {
target_id = <something unique>
rule = aws_cloudwatch_event_rule.step_function_event_rule.name
arn = <step function arn>
role_arn = <role that allows eventbridge to start execution on your behalf>
}
I'm having an issue when creating a bucket notification to trigger a Lambda function. The error:
Error putting S3 notification configuration: InvalidArgument: Unable to validate the following destination configurations
status code: 400
I've read that similar problems might be caused by the order in which the resources are created or that Lambda permissions are missing. However, I tried including depends_on in my code as well as applying the template couple of times and waiting in between. I'm using the least restrictive Lambda policy. I also tried using the exact sample code from the Terraform documentation, but that gives me a whole different error.
The exact same setup works fine if created in the console.
Here's the problematic part of my code:
resource "aws_lambda_function" "writeUsersToDB" {
filename = "writeUsersToDB.zip"
function_name = "writeUsersToDB"
role = "arn:aws:iam::0000000:role/AWSLambdaFullAccess"
handler = "main.lambda_handler"
memory_size = 256
timeout = 900
source_code_hash = filebase64sha256("writeUsersToDB.zip")
runtime = "python3.8"
environment {variables = local.parameters}
layers = [ "arn:aws:lambda:eu-west-2:0000000:layer:pandas-pandas-schema-numpy:1" ]
}
resource "aws_s3_bucket_notification" "event" {
bucket = aws_s3_bucket.user_data.id
lambda_function {
lambda_function_arn = aws_lambda_function.writeUsersToDB.arn
events = ["s3:ObjectCreated:*"]
filter_suffix = ".csv"
}
depends_on = [aws_lambda_function.writeUsersToDB]
}
resource "aws_s3_bucket" "user_data" {
bucket = "nameofthebucket"
}
You are missing aws_lambda_permission:
resource "aws_lambda_permission" "example" {
statement_id = "AllowExecutionFromS3Bucket"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.writeUsersToDB.function_name
principal = "s3.amazonaws.com"
source_arn = aws_s3_bucket.user_data.arn
}
In terraform , Trying to S3 bucket as trigger to my lambda and giving the permissions. For this use case , creating S3 resource and trying to refer that lambda function in triggering logic. But When I refer code is failing with below error.Please help me to resolve this issue .
#########################################
# Creating Lambda resource
###########################################
resource "aws_lambda_function" "test_lambda" {
filename = "output/welcome.zip"
function_name = var.function_name
role = var.role_name
handler = var.handler_name
runtime = var.run_time
}
######################################################
# Creating s3 resource for invoking to lambda function
######################################################
resource "aws_s3_bucket" "bucket" {
bucket = "source-bucktet-testing"
}
#####################################################################
# Adding S3 bucket as trigger to my lambda and giving the permissions
#####################################################################
resource "aws_s3_bucket_notification" "aws-lambda-trigger" {
bucket = aws_s3_bucket.bucket.id
lambda_function {
lambda_function_arn = aws_lambda_function.test_lambda.arn
events = ["s3:ObjectCreated:*"]
filter_prefix = "file-prefix"
filter_suffix = "file-extension"
}
}
resource "aws_lambda_permission" "test" {
statement_id = "AllowS3Invoke"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.test_lambda.function_name
principal = "s3.amazonaws.com"
source_arn = "arn:aws:s3:::aws_s3_bucket.bucket.id"
}
Error Message :
Error: Error putting S3 notification configuration: InvalidArgument: Unable to validate the following destination configurations
status code: 400, request id: 8D16EE1EF8FC0E63, host id: PlzqurwmHo3hDJdr0nUhOGuJKnghOBCtMImZ+8fEFX3JPjKV2M47UZuJ5Z26FalKxmoF1Xl8lag=
Your source_arn in aws_lambda_permission is incorrect. It should be:
source_arn = aws_s3_bucket.bucket.arn
At present your source_arn is literally string "arn:aws:s3:::aws_s3_bucket.bucket.id", which is incorrect.
This issue looks very much like a bug but I believe there must be something wrong in my terraform file because I can't find anybody on the web having the same problem.
Here is the part of my terraform file that creates a lambda and a topic rule for it:
resource "aws_lambda_function" "rds_persist" {
filename = "${local.rds_persist_file_path}"
function_name = "RdsPersist-${var.env}"
role = "${aws_iam_role.lambda_role.arn}"
handler = "package.handler"
source_code_hash = "${local.rds_persist_package_hash}"
runtime = "nodejs8.10"
memory_size = 128
timeout = 10
vpc_config = {
subnet_ids = ["${var.private_subnet_ids}"]
security_group_ids = ["${aws_security_group.all_vpc_access.id}"]
}
environment {
variables = {
DB = "${var.database_url}"
IOT_DEVICE_ARN = "${var.iot_device_v1_sns_arn}"
}
}
}
resource "aws_iot_topic_rule" "rds_push" {
name = "${var.env}_RdsPush"
description = "Pushes events to a persistence lambda (rds store)"
enabled = true
sql = "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), '${var.env}-')"
sql_version = "2016-03-23"
lambda {
function_arn = "${aws_lambda_function.rds_persist.arn}"
}
}
Here is the result in AWS Console:
If I delete and re-add the rule in the console, then the trigger appears on the lambda.
It might be the case your lambda function that your topic uses is being created before the function.
I tested this also by tainting the topic rule alone so it got recreated (see the logs below). Unfortunately it didn't solve the issue.
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
-/+ module.lambda.aws_iot_topic_rule.rds_push (tainted) (new resource required)
id: "dev_RdsPush" => <computed> (forces new resource)
arn: "arn:aws:iot:eu-west-1:827689093226:rule/dev_RdsPush" => <computed>
description: "Pushes events to a persistence lambda (rds store)" => "Pushes events to a persistence lambda (rds store)"
enabled: "true" => "true"
lambda.#: "1" => "1"
lambda.1860721139.function_arn: "arn:aws:lambda:eu-west-1:827689093226:function:RdsPersist-dev" => "arn:aws:lambda:eu-west-1:827689093226:function:RdsPersist-dev"
name: "dev_RdsPush" => "dev_RdsPush"
sql: "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), 'dev-')" => "SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM '#' WHERE startswith(clientid(), 'dev-')"
sql_version: "2016-03-23" => "2016-03-23"
Plan: 1 to add, 0 to change, 1 to destroy.
Update: I just found a very similar issue at a different place:
There's supposed to be an SNS subscription between this another lambda and a SNS. Here is the relevant code in the terraform:
resource "aws_sns_topic_subscription" "conference_call" {
topic_arn = "${module.sns.conference_call_arn}"
protocol = "lambda"
endpoint = "${module.lambda.messaging_arn}"
}
(Obviously I checked the resources and they are correct)
In the console I don't see the trigger in the lambda but I do see the subscription in SNS:
Update: exact same issue when creating the resources using AWS CLI
# For the first issue
$ aws iot create-topic-rule --rule-name dev_RdsPush --topic-rule-payload '{"sql":"SELECT * as payload, topic() as topic, timestamp() AS timestamp FROM \'#\' WHERE startswith(clientid(), \'dev-\')","actions":[{"lambda":{"functionArn":"arn:aws:lambda:eu-west-1:xxxxxxxxx:function:RdsPersist-dev"}}]}'
# For the second issue
$ aws sns subscribe --topic-arn arn:aws:sns:eu-west-1:xxxxxxxx:conference-call-v1-dev --protocol lambda --notification-endpoint arn:aws:lambda:eu-west-1:xxxxxxxxx:function:Messaging-dev
Solution:
Add these:
IoT:
resource "aws_lambda_permission" "conference_call_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.conference_call.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${var.conference_call_sns_arn}"
}
SNS:
resource "aws_lambda_permission" "messaging_sns" {
statement_id = "AllowExecutionFromSNS"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.messaging.function_name}"
principal = "sns.amazonaws.com"
source_arn = "${var.conference_call_sns_arn}"
}
You need to add a lambda permission to allow IoT to invoke lambda. The Lambda console uses the permissions of the function to show what can invoke it.
https://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.html#lambda-rule.
It might be the case your lambda function that your topic uses is being created before the function. Try adding depends_on = ["aws_lambda_function.rds_persist"] on your aws_iot_topic_rule and see how it goes.
In my case this was the missing permission:
resource "aws_lambda_permission" "rds_topic_rule_permission" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.rds_persist.function_name
principal = "iot.amazonaws.com"
source_arn = aws_iot_topic_rule.rds_push.arn
}
See: https://github.com/hashicorp/terraform-provider-aws/issues/24196#issuecomment-1103979956