I'm trying to create a ConfigurationSetEventDestination using serverless resources but it not recognizing the EventDestination for value SNSDestination, here is the output.
And here the resource from serverless.yml
resources:
Resources:
HandleEmailsEvents:
Type: AWS::SNS::Topic
Properties:
DisplayName: 'Handle emails events (${self:custom.stage})'
TopicName: 'HandleEmailsEvents'
ConfigurationSet:
Type: 'AWS::SES::ConfigurationSet'
Properties:
Name: 'EMAIL_TRACKING'
ConfigurationSetEventDestination:
Type: 'AWS::SES::ConfigurationSetEventDestination'
Properties:
ConfigurationSetName: 'EMAIL_TRACKING'
EventDestination:
Name: 'EMAIL_TRACKING_DESTINATION'
Enabled: true
MatchingEventTypes:
- bounce
- complaint
SNSDestination:
TopicARN:
Ref: 'HandleEmailsEvents'
Following the documentation ConfigurationSetEventDestination EventDestination seems to not be available, but here it is with this description object.
SNSDestination is also available when creating from console
#AWS What's going on here?
Thanks,
PS: I'm not the only one...
https://forums.aws.amazon.com/thread.jspa?messageID=858616󑧸
https://forums.aws.amazon.com/thread.jspa?messageID=809004󅠬
https://forums.aws.amazon.com/thread.jspa?messageID=848013󏂍
[UPDATED]
I tried creating the same via nodejs sdk, it works, its possible, docs here.
Could be something with serverless CloudFormation generated stack?
let ses = new AWS.SES()
const destinationParams = {
ConfigurationSetName: 'test',
EventDestination: {
Name: 'xxxxx2',
MatchingEventTypes: ['send', 'reject', 'bounce', 'complaint', 'delivery', 'open', 'click'],
Enabled: true,
SNSDestination: {
TopicARN: 'arn:aws:sns:us-east-1:xxxxxxxxxx:test',
},
},
};
ses.createConfigurationSetEventDestination(destinationParams, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
});
I found a solution, if you are using serverless framework.
1- Create a Resource to create the ConfigurationSet in serverless.yml
resources:
Resources:
ConfigurationSet:
Type: 'AWS::SES::ConfigurationSet'
Properties:
Name: 'EMAIL_TRACKING'
2- Install serverless-ses-sns
npm install --save-dev serverless-ses-sns
3- Add to serverless.yml
plugins:
- serverless-ses-sns
4- Finally add the configuration in serverless.yml
custom:
snsDestination:
region: <region> # If absent, self:provider.region will be used
configurationSet: 'EMAIL_TRACKING'
topicArn: <topic arn> # If absent, one will be created
events: # One or more of the following
- renderingFailure
- reject
- bounce
- send
- complaint
- delivery
- open
- click
Reference for the plugin
I also stumbled upon the same issue, so as of now, SNS cannot be passed as an event destination using cloudformation. For more info refer to the link, Do checkout the NOTE, its explicitly mentioned there.
Related
I want to trigger a lambda whenever a new EC2 instance is registred in SSM's Fleet Manager (meaning the instance can be connected to using SSM), however I can't find what pattern to use in EventBridge.
Within EventBridge, I tried using the following pattern I found in the docs (so far its looks like the closest thing to my goal):
{
"source": ["aws.ssm"],
"detail-type": ["Inventory Resource State Change"]
}
However when I create a new EC2 and wait for its SSM agent to become active, it still doesn't trigger the above pattern.
Any idea how to catch this kind of event?
I think you have to go through CloudTrail API call.
Please find below a CloudFormation template I used in the past that was working. Please note that it just provides the SSM resources. You need to add your own SQS queue as well (see SQS.ARN) and I've used the association with the tag registration set to enabled. So that if you have a lambda function connected, you can set it to false so if the instance connect again, it won't go to the same process again.
AWSTemplateFormatVersion: "2010-09-09"
Description: >
SSM Registration event
# Description of the resources to be created.
Resources:
RegistrationDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: Command
Content:
schemaVersion: "2.2"
description: >
An Automation Document ran by registered instances that gathers their software inventory
and automatically updates their AWS SSM Agent to the latest version.
mainSteps:
- name: GatherSoftware
action: aws:softwareInventory
- name: Sleep
action: aws:runShellScript
inputs:
runCommand:
- sleep 20 || true
- name: UpdateAgent
action: aws:updateSsmAgent
inputs:
agentName: amazon-ssm-agent
source: https://s3.{Region}.amazonaws.com/amazon-ssm-{Region}/ssm-agent-manifest.json
allowDowngrade: "false"
RegistrationDocumentAssociation:
Type: AWS::SSM::Association
Properties:
AssociationName: !Sub registration-association-${AWS::StackName}
Name: !Ref RegistrationDocument
Targets:
- Key: tag:registration
Values:
- enabled
RegistrationEventRule:
Type: AWS::Events::Rule
Properties:
Description: >
Events Rule that monitors registration of AWS SSM instances
and logs them to an SQS queue.
EventPattern:
source:
- aws.ssm
detail-type:
- AWS API Call via CloudTrail
detail:
eventName:
- UpdateInstanceAssociationStatus
requestParameters:
associationId:
- !Ref RegistrationDocumentAssociation
executionResult:
status:
- Success
State: ENABLED
Targets:
- Arn: SQS.ARN
Id: SqsRegistrationSubscription
SqsParameters:
MessageGroupId: registration.events
I have the following step function in my AWS SAM template, it was defined using the syntax in the documentation. I'm using intrinsic functions to get some pseudoparameters but something is going wrong with them.
SfdcOrderEventsStepFunction:
Type: AWS::Serverless::StateMachine
Properties:
DefinitionSubstitutions:
Region: !Ref "AWS::Region"
AccountId: !Ref "AWS::AccountId"
EventBusPublishTarget: "order-events"
DefinitionUri: sfn-definition.asl.yml
Events:
EventBridgeEvent:
Type: EventBridgeRule
Properties:
EventBusName: sfdc-events
Pattern:
# TODO: Update pattern when the salesforce service is ready
source:
- prefix: salesforce.com
detail-type:
- Order
detail:
Payload__c:
attributes:
type:
- order
InputPath: $.detail
Name: sfdc-order-events
Role: !Sub 'arn:aws:iam::${AWS::AccountId}:role/stepfunction_execution_role'
Tracing:
Enabled: true
when I try to deploy it shows me the following error:
Resource template validation failed for resource
SfdcOrderEventsStepFunction as the template has invalid properties.
Please refer to the resource documentation to fix the template.
Properties validation failed for resource SfdcOrderEventsStepFunction
with message:
#/De finitionSubstitutions/ AccountId: 2 subschemas matched instead of one
#/DefinitionSubstitutions/AccountId: expected type: Boolean, found: String
At the end it deploys without problems. The step function and all of its components run as expected and without errors, but I wanted to know if there if something I can do to fix the template.
I’m trying to setup an Amazon MSK cluster and connect to it from a lambda function. The lambda function will be a producer of messages, not a consumer.
I am using the serverless framework to provision everything and in my serverless.yml I have added the following and that seems to be working fine.
MSK:
Type: AWS::MSK::Cluster
Properties:
ClusterName: kafkaOne
KafkaVersion: 2.2.1
NumberOfBrokerNodes: 3
BrokerNodeGroupInfo:
InstanceType: kafka.t3.small
ClientSubnets:
- Ref: PrivateSubnet1
- Ref: PrivateSubnet2
- Ref: PrivateSubnet3
But when trying to connect to this cluster to actually send messages I am unsure how to get the connection string here? I presume it should be the ZookeeperConnectString?
I’m new to kafka/msk so maybe I am not seeing something obvious.
Any advice much appreciated. Cheers.
I don't know what kind of code base u are using, so I will add my code which I wrote in GO.
In essence you should connect to MSK cluster the same way as you would connect to some stand alone Kafka instance. We are using brokers for "connecting" or better said writing to MSK cluster.
I'm using segmentio/kafka-go library. My function for sending event to MSK cluster looks like this
// Add event
func addEvent(ctx context.Context, requestBody RequestBodyType) (bool, error) {
// Prepare dialer
dialer := &kafka.Dialer{
Timeout: 2 * time.Second,
DualStack: true,
}
brokers := []string{os.Getenv("KAFKA_BROKER_1"), os.Getenv("KAFKA_BROKER_2"), os.Getenv("KAFKA_BROKER_3"), os.Getenv("KAFKA_BROKER_4")}
// Prepare writer config
kafkaConfig := kafka.WriterConfig{
Brokers: brokers,
Topic: os.Getenv("KAFKA_TOPIC"),
Balancer: &kafka.Hash{},
Dialer: dialer,
}
// Prepare writer
w := kafka.NewWriter(kafkaConfig)
// Convert struct to json string
event, err := json.Marshal(requestBody)
if err != nil {
fmt.Println("Convert struct to json for writing to KAFKA failed")
panic(err)
}
// Write message
writeError := w.WriteMessages(ctx,
kafka.Message{
Key: []byte(requestBody.Event),
Value: []byte(event),
},
)
if writeError != nil {
fmt.Println("ERROR WRITING EVENT TO KAFKA")
panic("could not write message " + err.Error())
}
return true, nil
}
My serverless.yml
Upper code (addEvent) belongs to functions -> postEvent in serverless.yml... If you are consuming from kafka, then you should check functions -> processEvent. Consuming event is fairly simple, but setting everything up for producing to Kafka it crazy. We are probably working on this for month and a half and still figuring out how everything should be set up. Sadly serverless does not do everything for you, so you will have to "click trough" manually in AWS, but we compared to other frameworks and serverless is still the best right now
provider:
name: aws
runtime: go1.x
stage: dev
profile: ${env:AWS_PROFILE}
region: ${env:REGION}
apiName: my-app-${sls:stage}
lambdaHashingVersion: 20201221
environment:
ENV: ${env:ENV}
KAFKA_TOPIC: ${env:KAFKA_TOPIC}
KAFKA_BROKER_1: ${env:KAFKA_BROKER_1}
KAFKA_BROKER_2: ${env:KAFKA_BROKER_2}
KAFKA_BROKER_3: ${env:KAFKA_BROKER_3}
KAFKA_BROKER_4: ${env:KAFKA_BROKER_4}
KAFKA_ARN: ${env:KAFKA_ARN}
ACCESS_CONTROL_ORIGINS: ${env:ACCESS_CONTROL_ORIGINS}
ACCESS_CONTROL_HEADERS: ${env:ACCESS_CONTROL_HEADERS}
ACCESS_CONTROL_METHODS: ${env:ACCESS_CONTROL_METHODS}
BATCH_SIZE: ${env:BATCH_SIZE}
SLACK_API_TOKEN: ${env:SLACK_API_TOKEN}
SLACK_CHANNEL_ID: ${env:SLACK_CHANNEL_ID}
httpApi:
cors: true
apiGateway:
resourcePolicy:
- Effect: Allow
Action: '*'
Resource: '*'
Principal: '*'
vpc:
securityGroupIds:
- sg-*********
subnetIds:
- subnet-******
- subnet-*******
functions:
postEvent:
handler: bin/postEvent
package:
patterns:
- bin/postEvent
events:
- http:
path: event
method: post
cors:
origin: ${env:ACCESS_CONTROL_ORIGINS}
headers:
- Content-Type
- Content-Length
- Accept-Encoding
- Origin
- Referer
- Authorization
- X-CSRF-Token
- X-Amz-Date
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-Agent
allowCredentials: false
methods:
- OPTIONS
- POST
processEvent:
handler: bin/processEvent
package:
patterns:
- bin/processEvent
events:
- msk:
arn: ${env:KAFKA_ARN}
topic: ${env:KAFKA_TOPIC}
batchSize: ${env:BATCH_SIZE}
startingPosition: LATEST
resources:
Resources:
GatewayResponseDefault4XX:
Type: 'AWS::ApiGateway::GatewayResponse'
Properties:
ResponseParameters:
gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
gatewayresponse.header.Access-Control-Allow-Headers: "'*'"
ResponseType: DEFAULT_4XX
RestApiId:
Ref: 'ApiGatewayRestApi'
myDefaultRole:
Type: AWS::IAM::Role
Properties:
Path: /
RoleName: my-app-dev-eu-serverless-lambdaRole-${sls:stage} # required if you want to use 'serverless deploy --function' later on
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
# note that these rights are needed if you want your function to be able to communicate with resources within your vpc
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
- arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole
I must warn you that we spend a lot of time figuring out how to properly setup VPC and other networking / permission stuff. My collage will write blog post once he arrivers from vacation. :) I hope this helps you some how. Best of luck ;)
UPDATE
If you are using javascript, then you would connect to Kafka similar to this
const { Kafka } = require('kafkajs')
const kafka = new Kafka({
clientId: 'order-app',
brokers: [
'broker1:port',
'broker2:port',
],
ssl: true, // false
})
The connection string which is called broker bootstrap string can be found my making an API call like aws kafka get-bootstrap-brokers --cluster-arn ClusterArn
See example here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-get-bootstrap-brokers.html
Also here is a step by step walk through on how produce/consume data: https://docs.aws.amazon.com/msk/latest/developerguide/produce-consume.html
I got AWS sample serverless app in which I have single yaml files along with multiples handlers.
The problem is that template.yaml is keep growing, how can separate yaml for each handler or group of handlers so it would be easy to manage.
For our projects we started dividing the main YAML into multiple on in a next way:
all lambdas are still described in serverless.yml file
in serverless.yml
resources:
- ${file(./sls-config/cognito-user-pools-authorizer.yml)}
- ${file(./sls-config/aurora.yml)}
- ${file(./sls-config/bucket.yml)}
- ${file(./sls-config/queues.yml)}
- ${file(./sls-config/alarms.yml)}
- ${file(./sls-config/roles.yml)}
- ${file(./sls-config/outputs.yml)}
Yep, splitting of the resources up to you.
alarm.yml
Resources:
SQSAlarmTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: ${self:provider.prefix}-sqs-alarm-topic
TopicName: ${self:provider.prefix}-sqs-alarm-topic
Subscription:
- Endpoint: example-email#mail.com
Protocol: email
Tags: ${self:custom.sqsTags}
and so on.
outputs.yml
Outputs:
CognitoUserPoolId:
Value: ${self:custom.userPool}
CognitoUserPoolClientId:
Value: ${self:custom.userPoolClientId}
DSClusterID:
Description: "RDS Cluster "
Value: { Ref: RDSCluster }
DBAddress:
Value: !GetAtt RDSCluster.Endpoint.Address
The variables from custom and provide may be used easily through the sub_configs.yml.
Be careful with spacing/paddings in yaml files :)
I'm trying to deploy Step Function, but I see no ways to define activity in serverless config.
AWS docs https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-stepfunctions-activity.html saying that activity should be defined that way, but every time I do sls deploy I can't see new activity in step function console. Is it possible at all to create activity via serverless or I have to run script/create it manually?
Resources:
MyActivity:
Type: "AWS::StepFunctions::Activity"
Properties:
Name: myActivity
stepFunctions:
stateMachines:
stepfunctest:
events:
- http:
path: step
method: get
definition:
Comment: "A sample application"
StartAt: extract
States:
extract:
Type: Task
Resource: "arn:aws:state:#{AWS::Region}:#{AWS::AccountId}:activity:MyActivity"
End: true
assuming you're uing the serverless plugin https://github.com/serverless-operations/serverless-step-functions. You can create the activity by adding the activity into the stepFunction
stepFuntions:
activities:
- myActivity
stateMachines:
stepfunctest:
events:
- http:
path: step
method: get
definition:
Comment: "A sample application"
StartAt: extract
States:
extract:
Type: Task
Resource: "arn:aws:state:#{AWS::Region}:#{AWS::AccountId}:activity:MyActivity"
End: true