I am trying to set up a demo environment to try out SQS as an AWS Event Bridge Source. I tried uploading few documents to SQS to see if Event Bridge detects any change, but I don't see any events triggered. How can I test SQS as a source with AWS Event Bridge?
Resources:
Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ${AWS::StackName}
LambdaHandlerExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
EventConsumerFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.lambda_handler
Role: !GetAtt LambdaHandlerExecutionRole.Arn
Code:
ZipFile: |
import json
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=2))
Runtime: python3.7
Timeout: 50
EventRule:
Type: AWS::Events::Rule
Properties:
Description: eventEventRule
State: ENABLED
EventPattern:
source:
- aws.sqs
resources:
- !GetAtt Queue.Arn
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: EventConsumerFunctionTarget
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref EventConsumerFunction
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventRule.Arn
SQS data events (publishing new message) are not source events for Event Bridge (EB). Only management events can be picked up by EB, e.g.:
purging of the queue
creating of new queue
deletion of a queue
Also your event rule should be more generic for that:
EventRule:
Type: AWS::Events::Rule
Properties:
Description: eventEventRule
State: ENABLED
EventPattern:
source:
- aws.sqs
# resources:
# - !GetAtt Queue.Arn
Targets:
- Arn: !GetAtt EventConsumerFunction.Arn
Id: EventConsumerFunctionTarget
You can also enable CloudWatch trial and detect API events for the SQS. This should enable fetching more events.
I might be late but this can benefit someone else,
have a look at this:
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-ecs-patterns.QueueProcessingFargateService.html
This will handle scaling of the Fargate container based on a number of messages in the SQS Queue.
a simplest stack can be defined using AWS CDK as following:
queue = sqs.Queue(stack, "Queue")
cluster = aws_ecs.Cluster(
stack, 'FargateCluster'
)
queue_processing_fargate_service = QueueProcessingFargateService(stack, "Service",
cluster=cluster,
memory_limit_mi_b=512,
image=ecs.ContainerImage.from_registry("test"),
command=["-c", "4", "amazon.com"],
enable_logging=False,
desired_task_count=2,
environment={
"TEST_ENVIRONMENT_VARIABLE1": "test environment variable 1 value",
"TEST_ENVIRONMENT_VARIABLE2": "test environment variable 2 value"
},
queue=queue,
max_scaling_capacity=5,
container_name="test"
)
Related
I'm following this tutorial on enabling aws security hub with aws chatbot. I seem to be having an issue with deploying the custom resource named AddCustomActionSHResource1 in the following code, which is linked from the tutorial to here.
AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys CustomAction in SecurityHub to enable sending findings to Slack via AWS Chatbot
#==================================================
# Parameters
#==================================================
Parameters:
SlackWorkSpaceID:
Description: Slack workspace ID (Copy and Paste from AWS Chatbot Configured Clients Interface)
Type: String
MinLength: 9
MaxLength: 15
AllowedPattern: ^[a-zA-Z0-9_]*$
ConstraintDescription: |
Malformed Input Parameter: Environment must contain only upper and numbers. Length should be minimum of 9 characters and a maximum of 15 characters.
SlackChannelID:
Description: Slack Channel ID
Type: String
MinLength: 9
MaxLength: 15
AllowedPattern: ^[a-zA-Z0-9_]*$
ConstraintDescription: |
Malformed Input Parameter: Environment must contain only upper and numbers. Length should be a minimum of 9 characters and a maximum of 15 characters.
# CustomActionName:
# Description: Name of the Custom Action in SecurityHub
# Type: String
# AllowedPattern: ^[a-zA-Z0-9_]*$
# Default: Send_To_Slack
# #Default: 'Send To !Sub "${ChatApplication}"'
#==================================================
# Resources
#==================================================
Resources:
#======================================================
# Lambda Role to create Custom Action
#======================================================
LambdaIAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSLambdaExecute
Path: /
#======================================================
# Lambda Policy to create Custom Action
#======================================================
LambdaIAMPolicy:
Type: AWS::IAM::Policy
DependsOn: LambdaIAMRole
Properties:
PolicyName: LambdaCreateCustomActionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'securityhub:CreateActionTarget'
Resource: '*'
Roles:
- !Ref LambdaIAMRole
#======================================================
# Lambda Function to create Custom Action
#======================================================
LambdaCreateCustomAction:
Type: AWS::Lambda::Function
DependsOn: LambdaIAMPolicy
Properties:
FunctionName: addcustomactionsecurityhub
Description: CreateCustom Action in SecurityHub
Runtime: python3.7
Handler: index.lambda_handler
Code:
ZipFile: |
import boto3
import cfnresponse
def lambda_handler(event, context):
securityhub = boto3.client('securityhub')
response = securityhub.create_action_target(Name="Send_To_Slack",Description='Send Messages to ChatApplication via AWS ChatBot',Id='SendToSlack')
responseData = {}
responseData['Data'] = response
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")
MemorySize: 128
Timeout: 10
Role: !GetAtt LambdaIAMRole.Arn
#======================================================
# Custom Resource to Invoke the Lambda Function
#======================================================
AddCustomActionSHResource1:
Type: Custom::AddCustomActionSH
DependsOn: LambdaCreateCustomAction
Properties:
ServiceToken: !GetAtt LambdaCreateCustomAction.Arn
#======================================================
# SNS Topic
#======================================================
SNSTopicAWSChatBot:
Type: AWS::SNS::Topic
Properties:
DisplayName: AWS Chatbot SNS Topic
EventTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: topicPolicyCustomaction
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: 'sns:Publish'
Resource: '*'
Topics:
- !Ref SNSTopicAWSChatBot
#======================================================
# CloudWatch Event Rule
#======================================================
EventRuleCustomAction:
Type: AWS::Events::Rule
Properties:
Description: "SecurityHub Chatbot CustomAction"
EventPattern:
source:
- "aws.securityhub"
detail-type:
- "Security Hub Findings - Custom Action"
resources:
- !Sub 'arn:aws:securityhub:${AWS::Region}:${AWS::AccountId}:action/custom/SendToSlack'
State: "ENABLED"
Targets:
-
Arn:
Ref: "SNSTopicAWSChatBot"
Id: "OpsTopic"
ChatBotManageIAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "sns.amazonaws.com"
Action:
- "sts:AssumeRole"
SlackChannelConfig:
Type: AWS::Chatbot::SlackChannelConfiguration
Properties:
ConfigurationName: securityhubnotification
IamRoleArn: !GetAtt ChatBotManageIAMRole.Arn
LoggingLevel: NONE
SlackChannelId: !Ref SlackChannelID
SlackWorkspaceId: !Ref SlackWorkSpaceID
SnsTopicArns:
- !Ref SNSTopicAWSChatBot
When deploying this cloudformation file to cloudformation in AWS, everything deploys successfully except for AddCustomActionSHResource1. The error message is:
CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [18sa90d1-49s1-4as7-9fsc-b79ssd6csd9]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.
I found some information on updating the lambda function by just adding comments to it, but I don't believe that's the issue. Most of the useful information I found was from this aws link.
I have a subscription to a SNS topic that is configured to move a message to a DLQ if it can't be delivered successfully to a lambda function.
As described by this document, there are client-side and server-side errors. If a client-side error occurred the message is correctly moved to the DLQ but in case a server-side error occurred, the message is not moved to the DLQ. This document describes the delivery retries and the subscription does use the default delivery policy defined by the SNS topic. The retries do happen but after the retries are exhausted the message is not moved to the DLQ.
Now I wonder why the message is not moved correctly to the DLQ on server-side errors. Is there some more configuration missing?
I created the resources with the following AWS SAM template:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
lambda-test
Globals:
Function:
Timeout: 30
Resources:
EmailFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: EmailFunction
Handler: de.domain.email.App::handleRequest
Runtime: java11
Architectures:
- x86_64
MemorySize: 512
Environment:
Variables:
# https://aws.amazon.com/blogs/compute/optimizing-aws-lambda-function-performance-for-java/
JAVA_TOOL_OPTIONS: -XX:+TieredCompilation -XX:TieredStopAtLevel=1
EmailsTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: 'test-emails'
EmailFunctionInvokePermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref EmailFunction
Principal: sns.amazonaws.com
EmailDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: !Join ['', [!GetAtt EmailsTopic.TopicName, '-dlq']]
# Policy for DLQ: https://docs.aws.amazon.com/sns/latest/dg/sns-configure-dead-letter-queue.html
EmailDLQPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref EmailDLQ
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal: '*'
Action:
- 'sqs:GetQueueUrl'
- 'sqs:GetQueueAttributes'
- 'sqs:SetQueueAttributes'
- 'sqs:SendMessage'
- 'sqs:ReceiveMessage'
- 'sqs:DeleteMessage'
- 'sqs:PurgeQueue'
Resource:
- !GetAtt EmailDLQ.Arn
EmailsSubscription:
Type: AWS::SNS::Subscription
Properties:
TopicArn: !Ref EmailsTopic
Protocol: lambda
Endpoint: !GetAtt EmailFunction.Arn
RedrivePolicy:
deadLetterTargetArn: !GetAtt EmailDLQ.Arn
And the Java function just looks like this (and throws an exception when the message body is reject):
package de.domain.email;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.SNSEvent;
public class App implements RequestHandler<SNSEvent, Object> {
public Object handleRequest(final SNSEvent input, final Context context) {
input.getRecords().forEach(r -> {
context.getLogger().log(r.getSNS().getMessage() + "\n");
if (r.getSNS().getMessage().equals("reject"))
throw new IllegalStateException("reject");
});
return null;
}
}
I've recently started learning and implementing services using AWS services. So, I guess I'm missing some small steps which I can't figure it.
I'm trying to implement the following diagram using the Cloudformation template. Everything is working fine unless. The Lambda and SQS queue are subscribed to the SNS topic successfully. Whenever a file is stored at the bucket, or even when I publish a message to the SNS topic manually, the lambda function is triggered successfully, but the message is not published to the SQS queue. I've also added the AWS::SQS::QueuePolicy to allow SNS to send messages to SQS, but it still does not work.
template.yml:
...
Resources:
S3ObjectPutTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ${AppName}-vrp-creation-${Environment}-topic
BucketToSNSPermission:
Type: AWS::SNS::TopicPolicy
...
Bucket:
Type: AWS::S3::Bucket
...
Lambda:
Type: AWS::Serverless::Function
...
Queue:
Type: AWS::SQS::Queue
Properties:
DelaySeconds: 0
MaximumMessageSize: 262144
MessageRetentionPeriod: 864000
QueueName: !Sub ${AppName}-${Environment}-queue
ReceiveMessageWaitTimeSeconds: 0
VisibilityTimeout: 90
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
TopicToQueueSubscription:
Type: AWS::SNS::Subscription
Properties:
Protocol: sqs
TopicArn: !Ref S3ObjectPutTopic
Endpoint: !GetAtt Queue.Arn
RawMessageDelivery: true
The full Cloudformation template.yaml file: template.yaml
You have mentioned Service: s3.amazonaws.com instead of Service: sns.amazonaws.com in your SQS policy. Update the template and try.
TopicToQueuePermission:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref Queue
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: s3.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt Queue.Arn
Condition:
ArnEquals:
aws:SourceArn: !Ref S3ObjectPutTopic
Currently my serverless.yml file looks like this:
service: bbb
provider:
name: aws
runtime: go1.x
stage: dev
package:
exclude:
- ./**
include:
- ./bin/**
functions:
ccc:
handler: bin/executable
name: my1minutelambda
role:
'Fn::GetAtt':
- mylambdaexecutionrole
- Arn
resources:
Resources:
mylambdaexecutionrole:
Type: AWS::IAM::Role
Properties:
RoleName: my-basiclambdaexec-role
Description: This is my basiclambdaexecution role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
myminschedulerevent:
Type: AWS::Events::Rule
Properties:
Description: This is my 1 minute rate scheduler.
Name: my-1-min-trigger-event-scheduler
ScheduleExpression: rate(1 hour)
Targets:
-
Arn: "arn:aws:lambda:us-east-1:111111111111:function:my1minutelambda" #update your a/c Id
Id: "TargetFunctionV1"
command used to deploy: sls deploy
After deployment finished, I can see on aws management console that all my resources got created.
BUT I am not able to see cloudwatch trigger extablishment for my lambda function.
See below screenshot:
CloudWatch Event Rule created successfully. (Target section pointing to my lambda function)
Trigger link not established for my lambda:
Please let me know what i am missing here. Thank you.
Update#1:
After adding following lines (as suggested by Marcin), I am able to see "CloudWatch event".
EventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: my1minutelambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !GetAtt myminschedulerevent.Arn
But, I can't see CloudWatch logs!! So, I can't findout if my lambda function is executing. Please see image below:
I tried to replicate the issue using serverless framework.
To do so I added the following AWS::Lambda::Permission to the end of your template:
EventsPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: dsfgsdfg # <-- REPLACE this with your function name my1minutelambda
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt myminschedulrevent.Arn
After adding the permissions, the console showed the trigger as expected:
If all you are trying to do is get a Lambda function to execute on a schedule, the Serverless Framework already includes an event type expressly for that purpose:
functions:
crawl:
handler: crawl
events:
- schedule: rate(2 hours)
- schedule: cron(0 12 * * ? *)
It will set everything up for you with no need to add additional CloudFormation. You can find the documentation here: https://www.serverless.com/framework/docs/providers/aws/events/schedule/#schedule/
ScheduledRule:
Type: AWS::Events::Rule
Properties:
Name: "SphEvent"
Description: "ScheduledRule"
ScheduleExpression: "rate(1 hour)"
State: "ENABLED"
Targets:
- Arn: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:lambda-name"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "ScheduledRule"
- "Arn"
I'm writing AWS Lambda to read message from SQS and then save some messages to S3 after filtering. Since 80% messages will be filtered out, I decided write S3 once for 100000 SQS messages.
Is it possible to trigger the Lambda only when the messages in SQS reach 10000?
It's possible with help of AWS CloudWatch.
You could configure an AWS CloudWatch Alarm which triggers an "AlarmAction" as soon as your SQS queue got 100000 visible messages. In case of an "Alarm" you are notifying a SNS Topic which then triggers your AWS Lambda.
If you are using AWS CloudFormation it might look as the following:
AWSTemplateFormatVersion: 2010-09-09
Resources:
Queue:
Type: AWS::SQS::Queue
QueueVisibleMessagesTopic:
Type: AWS::SNS::Topic
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Lambda:
Type: AWS::Lambda::Function
Properties:
Handler: handler.index
Role: !GetAtt LambdaExecutionRole.Arn
Runtime: nodejs8.10
MemorySize: 128
Timeout: 10
LambdaSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt Lambda.Arn
Protocol: lambda
TopicArn: !Ref QueueVisibleMessagesTopic
LambdaSubscriptionPermissions:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt Lambda.Arn
Action: lambda:InvokeFunction
Principal: sns.amazonaws.com
SourceArn: !Ref QueueVisibleMessagesTopic
QueueVisibleMessagesAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
Namespace: AWS/SQS
MetricName: ApproximateNumberOfMessagesVisible
Dimensions:
- Name: QueueName
Value: !GetAtt Queue.QueueName
Statistic: Sum
Period: 300
EvaluationPeriods: 1
ComparisonOperator: GreaterThanOrEqualToThreshold
Threshold: 100000
AlarmActions:
- !Ref QueueVisibleMessagesTopic