I want to create an alert if something goes wrong with Lambda function especially when lambda throws an exception. I am planning to configure SNS topic to send a message if that alert is triggered.
All lambdas are created using CloudFormation scripts, so I am searching for a CloudFormation template to configure alarms on CloudWatch logs. I was not able to find a good/working sample. Sample code below .
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "AWS CloudTrail API Activity Alarm Template for CloudWatch Logs",
"Parameters" : {
"LogGroupName" : {
"Type" : "String",
"Default" : "CloudTrail/DefaultLogGroup",
"Description" : "Enter CloudWatch Logs log group name. Default is CloudTrail/DefaultLogGroup"
},
"Email" : {
"Type" : "String",
"Description" : "Email address to notify when an API activity has triggered an alarm"
}
},
"Resources" : {
"SecurityGroupChangesAlarm": {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"AlarmName" : "CloudTrailSecurityGroupChanges",
"AlarmDescription" : "Alarms when an API call is made to create, update or delete a Security Group.",
"AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }],
"MetricName" : "SecurityGroupEventCount",
"Namespace" : "CloudTrailMetrics",
"ComparisonOperator" : "GreaterThanOrEqualToThreshold",
"EvaluationPeriods" : "1",
"Period" : "300",
"Statistic" : "Sum",
"Threshold" : "1"
}
},
"AlarmNotificationTopic": {
"Type": "AWS::SNS::Topic",
"Properties": {
"Subscription": [
{
"Endpoint": { "Ref": "Email" },
"Protocol": "email"
}
]
}
}
}
}
In order to do this, we need to create a subscription filter on the log group for that lambda with FilterPattern: "Exception"
So whenever there is an Exception word in log message it will trigger a monitor lambda.
Following is a cloudformation template in YAML that I have written
Resources:
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: 'AllowLambdaAccess'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: "Allow"
Resource:
Fn::Join:
- ''
- - 'arn:aws:logs:'
- Ref: AWS::Region
- ':'
- Ref: AWS::AccountId
- ':log-group:/aws/lambda/*'
- Action:
- ec2:DescribeNetworkInterfaces
- ec2:CreateNetworkInterface
- ec2:DeleteNetworkInterface
Effect: "Allow"
Resource: "*"
RoleName: !Sub "${AWS::StackName}-LambdaExecutionRole"
SubscriptionFilter:
Type: "AWS::Logs::SubscriptionFilter"
DependsOn: "LambdaInvokePermission"
Properties:
LogGroupName: !Sub "/aws/lambda/${LogGroupName}"
FilterPattern: "Exception"
DestinationArn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
LambdaFunction:
Type: 'AWS::Lambda::Function'
Properties:
Code:
S3Bucket: !Ref S3BucketName
S3Key: !Ref ZipFile
Description: Monitor Lambda Function
Handler: 'index.handler'
MemorySize: 1536
Role: !GetAtt
- LambdaExecutionRole
- Arn
Runtime: nodejs6.10
Environment:
Variables:
SMTP_SERVER: !Ref SMTPServer
SMTP_PORT: !Ref SMTPPort
EMAIL_FROM: !Ref FromEmail
EMAIL_TO: !Ref ToEmail
Timeout: 300
FunctionName: !Sub "${AWS::StackName}-LambdaFunction"
VpcConfig:
SecurityGroupIds: !Split [ ",", !Ref SecurityGroupId ]
SubnetIds: !Split [ ",", !Ref SubnetIds ]
DependsOn:
- LambdaExecutionRole
LambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: !Sub "logs.${AWS::Region}.amazonaws.com"
SourceArn:
Fn::Join:
- ''
- - 'arn:aws:logs:'
- Ref: AWS::Region
- ':'
- Ref: AWS::AccountId
- !Sub ':log-group:/aws/lambda/${LogGroupName}*'
Related
I wanted to use the ARN as parameter input to cloudformation stack resources EventRuleRegion1 - Target as well as EventBridgeIAMrole , but it is not working. when i call with Ref function
Original ARN
arn:aws:events:ap-southeast-2:123456789123:event-bus/central-eventbus-sydney
When i give the arn directly in code its working fine.
Code
AWSTemplateFormatVersion: 2010-09-09
Parameters:
EventBridgeName:
Description: Enter the Event Bridge Name
Type: String
Default: ec2-lifecycle-events
EventBusName:
Description: Enter the Central Event Bus Name
Type: String
Default: central-eventbus-sydney
EventBusArn:
Description: Enter the ARN of Central Event Bus
Type: String
Default: arn:aws:events:ap-southeast-2:123456789123:event-bus/central-eventbus-sydney
Monitoringaccount:
Description: Enter the Monitoring AWS account number
Type: String
Default: 123456789123
Resources:
EventRuleRegion1:
Type: AWS::Events::Rule
Properties:
Description: Event rule to send events to monitoring account event bus
EventBusName: default
EventPattern:
source:
- aws.ec2
detail-type:
- "EC2 Instance State-change Notification"
detail:
state:
- "running"
- "stopped"
- "terminated"
Name: !Ref EventBridgeName
State: ENABLED
Targets:
- Arn: >-
- !Join [ "", [ !Sub "arn:aws:events:${AWS::Region}:123456789123:event-bus/",!Ref EventBusName ] ]
Id: !Ref EventBusName
RoleArn: !GetAtt
- EventBridgeIAMrole
- Arn
EventBridgeIAMrole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: !Sub events.amazonaws.com
Action: 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: PutEventsDestinationBus
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'events:PutEvents'
Resource:
- >-
- !Join [ "", [ !Sub "arn:aws:events:${AWS::Region}:123456789123:event-bus/",!Ref EventBusName ] ]
Error
Parameter - !Join [ "", [ !Sub "arn:aws:events:${AWS::Region}:123456789123:event-bus/",!Ref EventBusName ] ] is not valid. Reason: Provided Arn is not in correct format. (Service: AmazonCloudWatchEvents; Status Code: 400; Error Code: ValidationException; Request ID: 0d52a1d6-095e-44f7-9455-b7481dc4fb8d; Proxy: null)
The use of >- will result in literal strings, not evaluation of your CFN functions (join, ref). It should be:
Targets:
- Arn: !Join [ "", [ !Sub "arn:aws:events:${AWS::Region}:123456789123:event-bus/",!Ref EventBusName ] ]
Below is my Cloudformation template I want add multiple resources getting below error
Template contains errors.: Template format error: YAML not well-formed. (line 61, column 1)
AWSTemplateFormatVersion: 2010-09-09
Description: >-
This template creates IoT policy - attaches to a device certificate, IoT Topic
Rule- used to forward messages to sns based on service key, and creates
required IAM roles for these.
Parameters:
vpcname:
Type: String
Description: Enter vpcname
vpcnamefirstletterupper:
Type: String
Description: Enter vpcname with camelcase, ex- "Usdevms"
taaccountid:
Type: String
Description: Enter TA AccountID"
Resources:
IoTDaasDeviceRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ["",[IoTDaasDeviceRole.,!Ref vpcname]]
MaxSessionDuration : 43200
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
AWS: !Join ["",[!Sub 'arn:aws:iam::${AWS::AccountId}:role/Daas',!Ref vpcnamefirstletterupper,'IotCredentialLambda']]
Service: lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
IoTDaasDevicePolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
Description: >-
This Policy will be attached to the device role and lists the
permissions given to device certificates
ManagedPolicyName: !Join
- ''
- - 'IoTDaasDeviceConnectPolicy.'
- !Ref vpcname
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: 'iot:Connect'
Resource: !Join
- ''
- - !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:client/'
- '*'
- Effect: Allow
Action: 'iot:Publish'
Resource: !Join
- ''
- - !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/rules/daas_device_events_rule_'
- !Ref vpcname
- '/*'
- Effect: Allow
Action: 'iot:StartNextPendingJobExecution'
Resource: {
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:things/','*']],
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/things/thingName/jobs/start-next/']],
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/things/thingName/jobs/start-next/accepted']],
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/things/thingName/jobs/start-next/rejected']]
}
- Effect: Allow
Action: 'iot:UpdateJobExecution'
Resource: !Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:things/','*']]
- Effect: Allow
Action: 'execute-api:Invoke'
Resource: !Join ['', [!Sub 'arn:aws:execute-api:${AWS::Region}:',!Ref taaccountid,':hpe5n6k1v8/Test/GET']]
Roles:
- Ref: IoTDaasDeviceRole
The following is incorrect:
Resource: {
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:things/','*']],
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/things/thingName/jobs/start-next/']],
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/things/thingName/jobs/start-next/accepted']],
!Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/things/thingName/jobs/start-next/rejected']] }
as it creates a map, but you need a list:
Resource:
- !Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:things/','*']]
- !Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/$aws/things/thingName/jobs/start-next/']]
- !Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/things/thingName/jobs/start-next/accepted']]
- !Join ["",[!Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topicfilter/$aws/things/thingName/jobs/start-next/rejected']]
Please note that there still can be issues in your template, which are not apparent yet.
You can use the aws cloudformation command line to validate your template, here's the output when running it on the file you provided:
$ aws cloudformation validate-template --template-body file://test.template
An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: [/Resources/IoTDaasDevicePolicy/Type/PolicyDocument/Statement/2/Resource] map keys must be strings; received a map instead
The validation succeeded when I corrected the faulty bloc using Marcin's answer:
{
"Parameters": [
{
"ParameterKey": "vpcname",
"NoEcho": false,
"Description": "Enter vpcname"
},
{
"ParameterKey": "taaccountid",
"NoEcho": false,
"Description": "Enter TA AccountID\""
},
{
"ParameterKey": "vpcnamefirstletterupper",
"NoEcho": false,
"Description": "Enter vpcname with camelcase, ex- \"Usdevms\""
}
],
"Description": "This template creates IoT policy - attaches to a device certificate, IoT Topic Rule- used to forward messages to sns based on service key, and creates required IAM roles for these.",
"Capabilities": [
"CAPABILITY_NAMED_IAM"
],
"CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::Role]"
}
I am working with aws cli cloudformation. While using the JSON parameters file along with yml template, I keep getting the error. I tried using create stack update stack as well as the change set.
Error parsing parameter '--parameters': Expected: '=', received: 'P' for input:
- ParameterKey: FunctionName
^
ParameterValue: taskaplambda
- ParameterKey: MemorySize
ParameterValue: 512
- ParameterKey: Timeout
ParameterValue: 5
Where my command is:
aws cloudformation update-stack --stack-name apstack --template-body file://templates/cflambdatemplate.yaml --parameters file://params/param.json
And my param.json is:
[
{
"ParameterKey": "FunctionName",
"ParameterValue": "taskaplambda"
},
{
"ParameterKey": "MemorySize",
"ParameterValue": 512
},
{
"ParameterKey": "Timeout",
"ParameterValue": 5
}
]
This is my YAML file
cflambdatemplate.yaml
Transform: AWS::Serverless-2016-10-31
Resources:
tasklambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Ref FunctionName
Handler: lambda_function.lambda_handler
MemorySize: !Ref MemorySize
Role:
Fn::GetAtt:
- "tasklambdarole"
- "Arn"
Runtime: python3.7
Timeout: !Ref Timeout
CodeUri:
Bucket: taskapbucket
Key: apbuild/lambda_function.zip
tasklambdarole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
taskPolicies:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
Roles:
-
Ref: "tasklambdarole"
Parameters:
FunctionName:
Type: String
MinLength: '3'
MaxLength: '18'
MemorySize:
Type: Number
MinValue: '128'
MaxValue: '1024'
Timeout:
Type: Number
MinValue: '1'
MaxValue: '15'
I have been trying this with every possibility, but it keeps giving me an error.
Just able to create everything you needed through aws CLI:
I'm using the same yaml file and json template of parameter and not getting any error. Below is the only change that I did in param.json :
[
{
"ParameterKey": "FunctionName",
"ParameterValue": "taskaplambda"
},
{
"ParameterKey": "MemorySize",
"ParameterValue": "512"
},
{
"ParameterKey": "Timeout",
"ParameterValue": "5"
}
]
You need to convert the Number to String, It's because CloudFormation parameter types don't map to JSON types, so the CLI expects everything to be passed as string.
I'm creating an ASG group which has a lifecyclehook for termination:
LifecycleHook:
Type: AWS::AutoScaling::LifecycleHook
Properties:
AutoScalingGroupName: !Ref NodeGroup
DefaultResult: CONTINUE
HeartbeatTimeout: 60
LifecycleHookName: !Sub "${AWS::StackName}-lifecycle-hook"
LifecycleTransition: autoscaling:EC2_INSTANCE_TERMINATING
Now I create a lambda function as well:
LambdaCreation:
Type: "AWS::Lambda::Function"
Properties:
Handler: "lambda_function.lambda_handler"
Environment:
Variables:
aws_region : !Ref AWSRegion
Role: !GetAtt LambdaExecutionRole.Arn
Code:
S3Bucket: !Ref LambdaCodeBucket
S3Key: "lambda-functions/function.zip"
Runtime: "python3.6"
Timeout: 60
On cloudwatch events, i created a rule for said event:
CloudwatchEvent:
Type: AWS::Events::Rule
Properties:
Description: ASG scale-in event to lambda
EventPattern: {
"source": [
"aws.autoscaling"
],
"detail-type": [
"EC2 Instance-terminate Lifecycle Action"
],
"detail": {
"AutoScalingGroupName":
[
{
"Fn::ImportValue" :
{
"Fn::Sub" : "${RootStackName}-nodes-asg-name"
}
}
]
}
}
State: ENABLED
Targets:
-
Arn:
!GetAtt LambdaCreation.Arn
Id:
!Ref LambdaCreation
But the lambda is never triggered.
Now, on AWS console I don't see a trigger on the designer. But if i add manually a cloudwatch trigger for the created rule, it starts working...
Why is the trigger on the lambda side not created? What am I missing?
Thanks all!
I faced the exact same frustration. Only difference is that I was using terraform but that's irrelavant.
You are missing this:
{
"Type" : "AWS::Lambda::Permission",
"Properties" : {
"Action" : String,
"EventSourceToken" : String,
"FunctionName" : String,
"Principal" : String,
"SourceAccount" : String,
"SourceArn" : String
}
}
The reason the "manual way" works because it creates the trigger AND the permission. When you provision stuff using IaC tools like Cloudformation/terraform, you need to explicitly specify this Lambda permission object.
The below code snippet creates a lambda function and creates a cloudwatch event to trigger the lambda function with necessary privileges.
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: root
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- s3:ListBucket
Resource: !Join [ '', [ 'arn:aws:s3:::', !Ref LambdaS3Bucket ] ]
- Effect: Allow
Action:
- s3:GetObject
Resource: !Join [ '', [ 'arn:aws:s3:::', !Ref LambdaS3Bucket, '/*' ] ]
- Effect: Allow
Action:
- sts:GetCallerIdentity
Resource: '*'
LambdaFunction:
Type: "AWS::Lambda::Function"
Properties:
Description: "Lambda function"
FunctionName: !Ref LambdaFunctionName
Handler: !Ref LambdaHandler
Runtime: !Ref LambdaRuntime
Timeout: !Ref LambdaTimeout
MemorySize: !Ref LambdaMemorysize
Role: !GetAtt LambdaExecutionRole.Arn
Code:
S3Bucket: !Ref LambdaS3Bucket
S3Key: !Ref LambdaS3BucketKey
Environment:
Variables:
time_interval_in_hours: !Ref TimeIntervalInHours
DependsOn: LambdaExecutionRole
CleanupEventRule:
Type: AWS::Events::Rule
Properties:
Description: "Cloudwatch Rule"
ScheduleExpression: !Ref CloudwatchScheduleExpression
State: !Ref CloudWatchEventState
Targets:
- Arn: !Sub ${LambdaFunction.Arn}
Id: "CleanupEventRule"
DependsOn: LambdaFunction
LambdaSchedulePermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Sub ${LambdaFunction.Arn}
Principal: 'events.amazonaws.com'
SourceArn: !Sub ${CleanupEventRule.Arn}
DependsOn: LambdaFunction
Here is the error.
An error occurred: SNSTopicPolicy - Invalid parameter:
Policy Error: null (Service: AmazonSNS;
Status Code: 400; Error Code: InvalidParameter;
Request ID: 38a567df-2cff-50bf-8f0e-33a91775cc6e).
I cannot find a place to look into logs to tell me what parameter is missing.?.? I'm pretty sure I have everything that is required. I'm using this site and their api pages.
SNSTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: Aura main Topic
TopicName: ${file(./env.yml):${opt:stage, self:provider.stage}.env.auraSnsTopicName}
SNSTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: auraAllowSQSsendrobelrobel
Statement:
-
Effect: Allow
Action:
- sns: Publish
Resource: { "Fn::GetAtt":["SQSQueue" ,"Arn"]}
Principle:
AWS: "*"
Topics:
- { "Ref": "SNSTopic" }
SQSQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${file(./env.yml):${opt:stage, self:provider.stage}.env.eeegPagesQueueName}
RedrivePolicy:
deadLetterTargetArn: {"Fn::GetAtt" : [ "SQSQueueDLQ", "Arn" ]}
maxReceiveCount: 2
SQSQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Id: allowSNSSourceAndLambdaTrigger
Statement:
-
Effect: Allow
Action:
- SQS:ReceiveMessage
- SQS:SendMessage
- SQS:ChangeMessageVisibility
- SQS:ListDeadLetterSourceQueues
- SQS:GetQueueUrl
# - lambda:CreateEventSourceMapping
# - lambda:ListEventSourceMappings
# - lambda:ListFunction
Resource: {"Ref": "SNSTopic"}
Queues:
- { "Ref": "SQSQueue" }
I have an SQS Policy statement that is working just fine.
Your Environment Information ----------------------------
OS: Mac darwin
Node Version: 8.12.0
Serverless Version: 1.32.0
Turns out you need spelling is important and also the AWS sub object was not needed.
Principal: "*"
Working Policy below:
SNSTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Statement:
- Sid: auraAllowSQSPublish
Effect: Allow
Principal: "*"
Action: "sns:Publish"
Resource: { "Ref": "SNSTopic" }
Topics:
- { "Ref": "SNSTopic" }