I'm currently using SNS to fan out messages to a few SQS queues.
Trying to parse the SQS message and was wondering if there is an out of the box message model to parse the queue message body in to because it's a SNS notification.
Is there such out-of-the-box model available for this kind of scenario?
E.g.: For S3+SNS events -> com.amazonaws.services.s3.event.S3EventNotification
I checked the SQS/ SNS Java SDKs and couldn't find anything similar.
Does this mean the model has to be built in the application code to parse such message bodies?
E.g.:
In the below example queue message, "Body" section is a SNS notification where as I'm interested in "Body -> "Message" section. Can the "Body" be parsed in to an out-of-the-box AWS model in this case?
{
"Body": {
"Type": "Notification",
"MessageId": "272a7e6b-ea5e-46c3-991f-3563d7cd3f09",
"Token": null,
"TopicArn": "arn:aws:sns:us-east-1:000000000000:contact-subscription-topic",
"Message": {
"type": "contactSubInstantiationMessage"
},
"SubscribeURL": null,
"Timestamp": "2020-10-20T03:54:14.022Z",
"SignatureVersion": "1",
"Signature": "EXAMPLEpH+..",
"SigningCertURL": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-0000000000000000000000.pem"
},
"Attributes": {
"ApproximateFirstReceiveTimestamp": "1603166057169",
"SenderId": "AIDAIT2UOQQY3AUEKVGXU",
"ApproximateReceiveCount": "1",
"SentTimestamp": "1603166054040"
},
"ReceiptHandle": "ibetdkwxaxkqzjxhrkqtgtbrneyylminkvatzwcounxxnubhdktzzkdqrgzxqsebrdfuuxpwnhbuyhvrcbrwxbfgvgdekcygsgauxtcmouzzhlyqvaazkpqmvmmpixbhnpfpldlgjzcnkmaupbikegthoqvdmxyjcvetpisdzxpxrtsrtxvpbmyln",
"MD5OfBody": "9657ff8451167353e3d11c492d99d15f",
"MessageId": "879b6742-8006-bef5-d233-f7b8c8bb33d7"
}
Appreciate your thoughts on this.
Thanks team!
The following post describes a way to eliminate the above mentioned SNS metadata by setting "RawMessageDelivery" to "true" for the subscription;
Amazon SNS -> SQS message body
https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html
Marking the issue as resolved.
Thanks.
Related
I’ve already created a sns topic and sqs queues with filter policies set. Now I wanted to use this in step function.
So how can I use ?
I know we have to write message attributes in which filter policies were applied but I don’t know how to specify that in Amazon state language.?
Thanks for help.
As I’m new to Aws so don’t know much about this, so I’m learning.
I tried this-
{
"Type": "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"TopicArn": "arn:aws:sns:us-west-2:111122223333:MySNSTopic",
"Message": "{"transaction_no": "12345", "message_body": "This is a test message."}",
"Subject": "Test Message"
},
"Next": "NextState"
}
But this is not correct.
I wanted to send only those message to sqs which are having filter policy of transaction_no
Message": "{"transaction_no": "12345", "message_body": "This is a test message."}",
Is this correct way how we specify filter policies in Amazon state language?
For Amazon state language, specify the message attributes by using the parameter "MessageAttributes" because we are using a "transaction_no" filter policy as a message attribute this parameter will work. The "DataType" is the Data Type of the attribute and "StringValue" is the value of the attribute.
Sample code
{
"Comment": "Sending test message to SQS Queue",
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage",
"Parameters": {
"QueueUrl": "arn:aws:sqs:us-west-2:111122223333:MyTestSQSQueue",
"MessageBody": "This is a test message.",
"MessageAttributes": {
"transaction_no": {
"DataType": "String",
"StringValue": "12345"
}
}
},
"Next": "NextState"
}
I am using AWS amplify to host my react app. The source code is on GitHub.
I have a lambda listening to AWS SNS and pushes a message to Slack to inform about the deployment status.
I want to achieve that the Slack notification includes PR name (or commit message if it was directly pushed to the branch instead of merging the PR).
I cannot find the commit message/PR name within SNS message. Is it possible to get this information retrieved?
The SNS event I get looks like
{
"Records": [
{
"EventSource": "aws:sns",
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:xx-central-x:xxxxxxx:amplify-xxxxxx_main:aaaaaaaa-2189-bbbb-8ca3-cccccccccccc",
"Sns": {
"Type": "Notification",
"MessageId": "aaaaaaaa-107b-bbbb-a47b-cccccccccccc",
"TopicArn": "arn:aws:sns:xx-central-x:001700307000:amplify-xxxxxx_main",
"Subject": null,
"Message": "\"Build notification from the AWS Amplify Console for app: https://main.xxxxx.amplifyapp.com/. Your build status is STARTED. Go to https://link.com to view details on your build. \"",
"Timestamp": "2021-08-05T08:36:37.919Z",
"SignatureVersion": "1",
"Signature": "DaPEZM6+WumrAg==",
"SigningCertUrl": "https://sns.xx-central-x.amazonaws.com/SimpleNotificationService-xxxxx.pem",
"UnsubscribeUrl": "https://sns.xx-central-x.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:xx-central-x:xxxxx:amplify-xxxxxx_main:aaaaaaaa-2189-bbbb-8ca3-cccccccccccc",
"MessageAttributes": {}
}
}
]
}
What I'm trying to implement is that on the AWS billing alarm I want to send an email and invoke a lambda function written in python from where I want to update something in the database. So how can I get these both, I know that against alarm you can send an email to the user but how to do both when an alarm is triggered?
{
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"AlarmName": "AWS Polly Alarm One",
"AlarmDescription": "AWS Polly Alarm One",
"ActionsEnabled": true,
"OKActions": [],
"AlarmActions": [
"arn:aws:sns:us-east-1:XXXXXXXXXXXX:Turn_Off_AWS_Polly_Send_Email_One"
],
"InsufficientDataActions": [],
"MetricName": "EstimatedCharges",
"Namespace": "AWS/Billing",
"Statistic": "Maximum",
"Dimensions": [
{
"Name": "ServiceName",
"Value": "AmazonPolly"
},
{
"Name": "Currency",
"Value": "USD"
}
],
"Period": 60,
"EvaluationPeriods": 1,
"DatapointsToAlarm": 1,
"Threshold": 0,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"TreatMissingData": "missing"
}
}
When configuring an Amazon CloudWatch alarm, you can specify an Amazon Simple Notification Service (SNS) topic. Messages relating to the alarm will be sent to this topic.
Amazon SNS is a 'publish-subscribe' model, meaning that recipients need to subscribe to the topic to receive a notification. Subscription types include:
Email
AWS Lambda functions
All subscribers to a topic will receive messages sent to the topic.
When a subscribed AWS Lambda function receives a message from an SNS topic, it is provided with details of the alarm that triggered the message. You can use this information to store relevant data in the database.
I have integrated pagerduty with AWS cloudwatch and I am trying to publish a message manually to a SNS Topic that is subscribed by pagerduty and email. But I am not able to get incidents in pagerduty. However, cloudwatch alarms are triggering incidents in pagerduty using this same topic.
I referred some document for pagerduty message payload. But unable to make it work. My SNS message JSON is as follows,
{
"default":"test message",
"email":"test email message",
"https":{
"service_key":"XXXX",
"event_type":"trigger",
"description":"Example alert on host1.example.com"
}
}
Its not triggering an incident in pagerduty. I am not sure what I am missing in the request body. I am receiving email messages properly from this same message body. Could someone point out the mistake?
Thanks in advance.
To do so, you must choose the option Custom Event Transformer for the PagerDuty Integration. In the integration, you can write your own JavaScript code as follows:
var normalized_event = {
event_type: PD.Trigger,
description: "SNS Event",
details: PD.inputRequest
};
PD.emitGenericEvents([normalized_event]);
To parse the received payload from SNS, you can use:
var rawBody = PD.inputRequest.rawBody;
var obj = JSON.parse(unescape(rawBody));
And treat obj to treat your event according to your SNS message.
I believe PagerDuty's native AWS CloudWatch integration is opinionated. So a Custom SNS message won't trigger an incident.
But PagerDuty has an inbound integration type that allows you to create a script using JS (ES5) to parse any custom message sent to the this integration - which can then trigger an incident based on the logic of your script.
Docs on the Custom Event Transformer: https://v2.developer.pagerduty.com/docs/creating-an-integration-inline
I'm too late to answer this but still adding as #filipebarretto has suggested we need to use Custom Event Transformer for this type of integration.
Setup: ~ AWS Cloudwatch (RDS Metric) -> AWS SNS -> PagerDuty (CET)
I have successfully integrated AWS SNS to PagerDuty via Custom Event Transformer
var body = JSON.parse(PD.inputRequest.rawBody)
var message = body.NewStateReason
var normalized_event = {
event_type: PD.Trigger,
description: body.AlarmName,
details: message
};
PD.emitGenericEvents([normalized_event]);
The above code will send incident as AlarmName and details as NewStateReason.
I tested with below sample events as SNS message, it works fine.
{
"version": "0",
"id": "bba1bcef-5268-9967-8628-9a6d09e042e9",
"detail-type": "CloudWatch Alarm State Change",
"source": "aws.cloudwatch",
"account": "[Account ID]",
"time": "2020-11-17T06:25:42Z",
"region": "[region Id]",
"resources": [
"arn:aws:cloudwatch:[region Id]:[Account ID]:alarm:CPUUtilize"
],
"detail": {
"alarmName": "CPUUtilize",
"state": {
"value": "ALARM",
"reason": "Threshold Crossed: 1 out of the last 1 datapoints [4.314689265544354 (17/11/20 06:20:00)] was less than the threshold (70.0) (minimum 1 datapoint for OK -> ALARM transition).",
"reasonData": {
"version": "1.0",
"queryDate": "2020-11-17T06:25:42.491+0000",
"startDate": "2020-11-17T06:20:00.000+0000",
"statistic": "Average",
"period": 300,
"recentDatapoints": [
4.314689
],
"threshold": 70
},
"timestamp": "2020-11-17T06:25:42.493+0000"
},
"previousState": {
"value": "OK",
"reason": "Threshold Crossed: 1 out of the last 1 datapoints [4.484088172640544 (17/11/20 05:44:00)] was not greater than or equal to the threshold (70.0) (minimum 1 datapoint for ALARM -> OK transition).",
"reasonData": {
"version": "1.0",
"queryDate": "2020-11-17T05:49:53.688+0000",
"startDate": "2020-11-17T05:44:00.000+0000",
"statistic": "Average",
"period": 300,
"recentDatapoints": [
4.484088
],
"threshold": 70
},
"timestamp": "2020-11-17T05:49:53.691+0000"
},
"configuration": {
"description": "Alarm Notification in my local timezone",
"metrics": [
{
"id": "16baea70-421b-0a6e-f6f1-bc913d2bf647",
"metricStat": {
"metric": {
"namespace": "AWS/EC2",
"name": "CPUUtilization",
"dimensions": {
"InstanceId": "i-0e448XXXXXXXXXXXX"
}
},
"period": 300,
"stat": "Average"
},
"returnData": true
}
]
}
}
}
Took from https://aws.amazon.com/blogs/mt/customize-amazon-cloudwatch-alarm-notifications-to-your-local-time-zone-part-1/
I am even later to the game here, but ...
How are you 'manually' sending the events? Did you check that the Policy on the SNS topic allows publishing of notifications from whichever service you are using to publish the events?
I had a similar issue with publishing notifications/events from AWS Backup. I had to add something like this to the Access Policy:
{
"Sid": "My-statement-id",
"Effect": "Allow",
"Principal": {
"Service": "backup.amazonaws.com"
},
"Action": "SNS:Publish",
"Resource": "arn:aws:sns:region:account-id:myTopic"
}
I have stumbled upon a rather strange and, to my knowledge, undocumented behaviour of Amazon SNS. I am looking for a solution or settings to fix it.
SUMMARY
I have a SNS Topic, with an HTTPS subscription pointing to an Amazon API Gateway REST endpoint, backed with a Node.js Lambda function for executing the request.
Now, if I use SNS and Publish on the topic , the whole API Gateway mapping template gets ignored/short-circuited. The Lambda function ends-up receiving ONLY the original SNS JSON object.
However, if I use a web browser (or curl) to access the endpoint the API Gateway Mapping Translation gets called and the proper JSON data gets passed onto the Lambda function.
THE API Gateway Endpoint
The API Gateway (aka TheApi hereafter) is created with a sms resource under which a "path parameters" {phone}. Therefore, you can query https://TheApi/sms/111-222-3333 with either a POST or GET method.
Both methods have a generic Mapping Template which grabs all paths parameters, all headers parameters, all query parameters and the whole request body and translate that into one LARGE request body JSON object. Here is what the template looks like:
{
"resource-path" : "$context.resourcePath",
"http-method" : "$context.httpMethod",
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
},
"query": {
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
},
"paths": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
},
"body" : $input.json('$')
}
This resulting object is then fed to the Lambda function as the event on which the lambda function operates. Here is the result of a simple API Gateway "Test":
Tue Feb 09 00:54:13 UTC 2016 : Endpoint request body after transformations:
{
"resource-path" : "/sms/{phone}",
"http-method" : "POST",
"headers": {
},
"query": {
},
"paths": {
"phone": "111-222-3333"
},
"body" : {"foo":"bar","Alice":"Bob"}
}
This endpoint and the called Lambda function work flawlessly when called from a Web Browser (or a curl call ). AWS Cloud Watch logs show that everything is good under the sun, the Lambda event received is the same as above, therefore the Mapping translation is called.
THE Problem
Now, if I use SNS and Publish on the topic (the one with the HTTPS subscription on the API Gateway endpoint listed at the top), the whole API Gateway mapping template gets ignored/short-circuited.
The Lambda function ends-up receiving ONLY the original SNS JSON object and none of that custom mapping I wrote. The Lambda function doesn't receive any information about the calling agent, the requested url, the headers.... nada!! Here what the Lambda event looks like, as shown in CloudWatch:
{
"Type": "Notification",
"MessageId": "d38077e1-406a-5122-8a57-38cecfc635fd",
"TopicArn": "arn:aws:sns:us-east-1:...:...",
"Subject": "Ceci est un test",
"Message": "Ceci est un message de test.",
"Timestamp": "2016-02-06T06:06:36.649Z",
"SignatureVersion": "1",
"Signature": "...",
"SigningCertURL": "...",
"MessageAttributes": {
"AWS.SNS.MOBILE.MPNS.Type": {
"Type": "String",
"Value": "token"
},
"AWS.SNS.MOBILE.MPNS.NotificationClass": {
"Type": "String",
"Value": "realtime"
},
"AWS.SNS.MOBILE.WNS.Type": {
"Type": "String",
"Value": "wns/badge"
}
}
}
As it can be seen, this JSON object is completely different.
FOOD for thoughts
Some may wonder: "Why I go to the trouble of making the API Gateway when I could forward SNS events directly to the Lambda functions?". The reason is quite simple, I need to attach additional information with the SNS message, in this case the phone number to send the message to. Using API Gateway I can create as many subscriptions to as many phone numbers without duplicating any code.
Others may wonder: "Why not using the SMS subscription built into SNS instead of making my own?". For one thing, I'm in Canada and Amazon SMS subscriptions no longer works in Canada. Secondly, I may wish to use another SMS service that Amazon's.
As it turns out, SNS topics can call Lambda functions directly. In which case, the SNS JSON object is exactly the same. Thus, it is as though AWS is detecting the HTTPS endpoint domain, resolving the underlying Lambda function and routing the call directly to the Lambda function without passing through the API Gateway services.
In fact, when I build another REST endpoint on another domain I control, I indeed receive the POST request with the SNS JSON body, which I can forward to the API Gateway endpoint and it gets translated just well.
Just like this:
{
"resource-path": "/sms/{phone}",
"http-method": "POST",
"headers": {
"Accept": "*/*",
"CloudFront-Forwarded-Proto": "https",
"CloudFront-Is-Desktop-Viewer": "true",
"CloudFront-Is-Mobile-Viewer": "false",
"CloudFront-Is-SmartTV-Viewer": "false",
"CloudFront-Is-Tablet-Viewer": "false",
"CloudFront-Viewer-Country": "US",
"Content-Type": "application/json",
"Via": "1.1 c903e93e57c533ecd52152e4407a295e.cloudfront.net (CloudFront)",
"X-Amz-Cf-Id": "Fy_dCf5yJbW1GOZWJMVJqhbz1qt6sLfNO0N33FqAtf56X1tB4py8Ig==",
"X-Forwarded-For": "69.65.27.156, 54.182.212.5",
"X-Forwarded-Port": "443",
"X-Forwarded-Proto": "https"
},
"query": {},
"paths": {
"phone": "14184901585"
},
"body": {
"Type": "Notification",
"MessageId": "d38077e1-406a-5122-8a57-38cecfc635fd",
"TopicArn": "arn:aws:sns:us-east-1:...:...",
"Subject": "Ceci est un test",
"Message": "Ceci est un message de test.",
"Timestamp": "2016-02-06T06:06:36.649Z",
"SignatureVersion": "1",
"Signature": "...",
"SigningCertURL": "...",
"UnsubscribeURL": "...",
"MessageAttributes": {
"AWS.SNS.MOBILE.MPNS.Type": {
"Type": "String",
"Value": "token"
},
"AWS.SNS.MOBILE.MPNS.NotificationClass": {
"Type": "String",
"Value": "realtime"
},
"AWS.SNS.MOBILE.WNS.Type": {
"Type": "String",
"Value": "wns/badge"
}
}
}
}
CALL for help
Are there any hidden settings somewhere when I can make this SNS -> API Gateway -> Lambda work with the proper Mapping Translation?
The mapping template is applied based on the request's content type. If there is no content type specified in the request, it defaults to 'application/json'.
Based on your description, I would assume that your mapping template is set up with the content type 'application/json'. This works fine as long as the client doesn't specify a different content type in its request (which is the case for browsers for example).
Since SNS sends the request with a header 'Content-type: text/plain' (SNS Send Message Over HTTP), it doesn't match your mapping template's content type and hence will ignore it. To get working you could either change the content type in your current mapping or add another one which matches 'text/plain'.
For some more details you could also have a look here in the AWS forums: Default Content-Type for Mapping Template
Best,
Jurgen