How to add SQS message attributes in SNS subscription? - amazon-web-services

The documentation for AWS SNS and SQS have sections about message attributes. But there is no explanation how to have SQS message attributes when that queue is subscribed to a SNS topic.
Is there a way to configure AWS SNS to add particular message attributes to the SQS messages send via the subscription?

From aws documentation:
To use message attributes with Amazon SQS endpoints, you must set the subscription attribute, Raw Message Delivery, to True. For more information about raw message delivery, see Appendix: Large Payload and Raw Message Delivery.
https://docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html
https://docs.aws.amazon.com/sns/latest/dg/large-payload-raw-message.html
Added an example from real-life project. Hopefully it helps to clarify things.
The message published to the sns topic is following:
aws sns publish --topic-arn arn:aws:sns:us-west-2:xxx:pollution-event --message '{"operatorId":3375001,"eventTypeId":1,"eventLevelId":1,"validFrom":"2018-03-10T09:00:00Z","validTo":"2018-03-11T09:00:00Z"}' --message-attributes '{"Type" : { "DataType":"String", "StringValue":"Orchestration.Services.Model.Pollution.PollutionMessage"}}'
Enable Raw Delivery is false (default). The message received by sqs contains only content, there are no attributes
{
"Type": "Notification",
"MessageId": "78d5bc6f-142c-5060-a75c-ef29b774ec66",
"TopicArn": "arn:aws:sns:eu-west-2:xxx:pollution-event",
"Message": "{\"validFrom\": \"2018-03-10T09:00:00Z\",\"validTo\": \"2018-03-11T09:00:00Z\",\"eventLevelId\": 1,\"eventTypeId\": 1,\"operatorId\": 3375001}",
"Timestamp": "2018-04-17T11:33:44.770Z",
"SignatureVersion": "1",
"Signature": "xxx==",
"SigningCertURL": "https://sns.eu-west-2.amazonaws.com/SimpleNotificationService-xxx.pem",
"UnsubscribeURL": "https://sns.eu-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-2:xxx",
"MessageAttributes": {
"Type": {
"Type": "String",
"Value": "Orchestration.Services.Model.Pollution.PollutionMessage"
},
"AWS.SNS.MOBILE.MPNS.Type": {
"Type": "String",
"Value": "token"
},
"AWS.SNS.MOBILE.MPNS.NotificationClass": {
"Type": "String",
"Value": "realtime"
},
"AWS.SNS.MOBILE.WNS.Type": {
"Type": "String",
"Value": "wns/badge"
}
}
}
Enable Raw delivery is true. The message contains the message attributes and proper content

Note: See other answer for better response, using Raw Message Delivery
While the Using Amazon SNS Message Attributes documentation sending Amazon SNS message attributes to Amazon SQS, it appears that the attributes are sent in the body of the message rather than being attached as message attributes on the resulting Amazon SQS messages.
For example, I did the following:
Created an Amazon SNS topic
Created an Amazon SQS queue and subscribed it to the SNS topic
Published a message to SNS
I published via the AWS Command-Line Interface (CLI):
aws sns publish --topic-arn arn:aws:sns:ap-southeast-2:123456789012:foo --message msg --subject subj --message-attributes '{"somename" : { "DataType":"String", "StringValue":"somevalue"}}'
(I got syntax help from map datatype in aws cli)
The resulting message in SQS showed the attributes as part of the message:
{
"Type" : "Notification",
"MessageId" : "53e3adad-723a-5eae-a7b7-fc0468ec2d37",
"TopicArn" : "arn:aws:sns:ap-southeast-2:123456789012:foo",
"Subject" : "subj",
"Message" : "msg",
"Timestamp" : "2017-05-29T12:48:22.186Z",
...
"MessageAttributes" : {
"somename" : {"Type":"String","Value":"somevalue"}
}
}
It would be much nicer if these attributes were attached to the SQS message as official SQS attributes. Alas, it seems this is not the case.

Enabled Raw message delivery type while adding SQS subscription for the topic inside SNS

If you are here because you have a SQS queue that is subscribed to an SNS topic, you checked that your subscription has set the Raw Message Delivery to True but you still cannot read an attribute on your SQS message:
Make sure that your SQS client is not filtering out message attributes.
The code below will only include myAttribute when receiving messages from the SQS queue:
SQS.receiveMessage({
QueueUrl: queueUrl,
VisibilityTimeout: 20,
WaitTimeSeconds: 10,
MessageAttributeNames: [
"myAttribute"
],
},...
If you want to read the value of some attribute other than myAttribute you will have to specify it (white list) or replace "myAttribute" with "All" to include all SQS attributes.
SQS.receiveMessage({
MessageAttributeNames: [
"myAttribute", "myOtherAttribute"
],
},...
Reference: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

Related

AWS Chatbot and EventBridge for Glue Job State Changes Error - Event received is not supported

I am trying to set up the AWS Chatbot with Slack integrations to display error messages for changes in states (errors) for AWS Glue. I have set up AWS EventBridge event pattern to catch Glue Job State Changes as follows:
{
"source": ["aws.glue"],
"detail-type": ["Glue Job State Change"],
"detail": {
"state": [
"FAILED"
]
}
}
This successfully catches all failed Glue Jobs and I have set up an AWS SNS topic as the target using the input transformer.
Input Transformer Input Path
{"jobname":"$.detail.jobName","jobrunid":"$.detail.jobRunId","jobstate":"$.detail.state"}
Input Transformer Input Template
"{\"detail-type\": \"Glue Job <job-name> has entered the state <job-state> with the message <message>.\"}"
AWS SNS has a subscriptions endpoint to the AWS Chatbot which fails to send the notification to Slack.
AWS Chatbot CloudWatch logs after an event using Input Transformer
Event received is not supported (see https://docs.aws.amazon.com/chatbot/latest/adminguide/related-services.html ):
{
"subscribeUrl": null,
"type": "Notification",
"signatureVersion": "1",
"signature": <signature>,
"topicArn": <topic-arn>,
"signingCertUrl": <signing-cert-url>,
"messageId": <message-id>,
"message": "{\"detail-type\": \"Glue Job MyJob has entered the state FAILED with the message SystemExit: None.\"}",
"subject": null,
"unsubscribeUrl": <unsubscribe-url>,
"timestamp": "2022-03-02T12:17:16.879Z",
"token": null
}
When the input is set to 'Matched Events' in the AWS EventBridge Select Target, the Slack Notification will send however it lacks any details.
Slack Notification
Glue Job State Change | eu-west-1 | Account: <account>
Glue Job State Change
AWS EventBridge Matched Events JSON Output
{
"Type" : "Notification",
"MessageId" : <message-id>,
"TopicArn" : <topic-arn>,
"Message" : "{\"detail-type\": [\"Glue Job State Change\"]}",
"Timestamp" : "2022-03-02T11:17:52.443Z",
"SignatureVersion" : "1",
"Signature" : <signature>,
"SigningCertURL" : <signing-cert-url>,
"UnsubscribeURL" : <unsubscribe-url>
}
There are very little differences between the two JSON outputs however the input transformer is considered an unsupported event. Is it possible to generate a custom message when using the AWS Chatbot for errors?
The best solution was to create a Lambda function as the target of the AWS EventBridge which performs a POST to a Slack Webhook.
# Import modules
import logging
import json
import urllib3
# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Define Lambda function
def lambda_handler(event, context):
http = urllib3.PoolManager()
url = <url>
link = <glue-studio-monitoring-link>
message = f"A Glue Job {event['detail']['jobName']} with Job Run ID {event['detail']['jobRunId']} has entered the state {event['detail']['state']} with error message: {event['detail']['message']}. Visit the link for job monitoring {link}"
logger.info(message)
headers = {"Content-type": "application/json"}
data = {'text': message}
response = http.request('POST',
url,
body = json.dumps(data),
headers = headers,
retries = False)
logger.info(response.status)

Amazon SNS not sending sms after account creation

I tried sending sms with amazon sns and I got
SMS message published to phone number +23481XXXXXXX successfully.
Message "ID": *********************
But the message status show "fail".
Please is there any other thing I have to do after verifying my account before I start sending sms?
You might also want to check if the region you're using supports sms and if the country you're trying to send has support.
https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html
Not sure if this is just SNS related or if you're trying to send sms after an account was created in cognito, might want to check region support for that too
https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-email-phone-verification.html
You can activate Delivery status logging to obtain a more-detailed log for SMS messages.
From Viewing Amazon CloudWatch metrics and logs for SMS deliveries - Amazon Simple Notification Service:
Activate Delivery status logging
Send a message
View the results in Amazon CloudWatch Logs
Here is an example log entry:
{
"notification": {
"messageId": "1077257a-92f3-5ca3-bc97-6a915b310625",
"timestamp": "2016-06-28 00:40:34.559"
},
"delivery": {
"mnc": 0,
"destination": "+1XXX5550100",
"priceInUSD": 0.00645,
"smsType": "Transactional",
"mcc": 0,
"providerResponse": "Unknown error attempting to reach phone",
"dwellTimeMs": 1420,
"dwellTimeMsUntilDeviceAck": 1692
},
"status": "FAILURE"
}

How to exclude cloudwatch alarns from SNS topic using filter policy?

I have one SNS topic and it has two subscriptions, one is pagerduty API and other is pagerdutyEmail.
I want to exclude alarms from pagerduty Email subscription using subscription filter policy for that topic.
I am not sure how I should exclude it, Does filter policy has like option
I tried to use below but doesnt work.
"AlarmArn": [
{
"prefix": "arn:aws:cloudwatch"
}
]
}
I don't think that the Subscription filter policy would allow you to filter the Cloudwatch Alarms based on Alarm ARN or any other metadata that is sent in the Cloudwatch JSON payload.
The filter policy expects a MessageAttributes field in the payload sent to the SNS topic, and that field should contain the attributes based on which you would like to add your filter policy to filter your notifications.
The JSON payload sent by a Cloudwatch Alarm doesn't contain those attributes as can be seen in the sample payload below:
{
"Type": "Notification",
"MessageId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"TopicArn": "arn:aws:sns:eu-west-1:000000000000:cloudwatch-alarms",
"Subject": "ALARM: \"Example alarm name\" in EU - Ireland",
"Message": "{\"AlarmName\":\"Example alarm name\",\"AlarmDescription\":\"Example alarm description.\",\"AWSAccountId\":\"000000000000\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 datapoint (10.0) was greater than or equal to the threshold (1.0).\",\"StateChangeTime\":\"2017-01-12T16:30:42.236+0000\",\"Region\":\"EU - Ireland\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"DeliveryErrors\",\"Namespace\":\"ExampleNamespace\",\"Statistic\":\"SUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":1.0}}",
"Timestamp": "2017-01-12T16:30:42.318Z",
"SignatureVersion": "1",
"Signature": "Cg==",
"SigningCertUrl": "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.pem",
"UnsubscribeUrl": "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:000000000000:cloudwatch-alarms:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
I've tested a filter policy using one of the key-value pairs available in the Cloudwatch payload but it didn't work for me.
Refer Doc:
https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html
You may have to write a Lambda Function using your preferred client library and use Pagerduty Events API to work out a solution.
Refer:
https://developer.pagerduty.com/docs/tools-libraries/client-libraries/
https://developer.pagerduty.com/docs/events-api-v2/trigger-events/
The filter above actually matches the messages with the AlarmArn with prefix arn:was:cloudwatch.
You might try to use anything-but command.
"AlarmArn": [{
"anything-but": [{
"prefix": "arn.aws.cloudwatch"
}]
}]
See https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html for more information
As of today, SNS supports payload-based message filtering too. Therefore, you can now filter the events coming from your CloudWatch alarms. More info: https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-sns-payload-based-message-filtering/

AWS SES sends same 'bounceRecipients' through SNS mulitple times

after the usual hair pulling efforts, I finally made AWSSimple Email Service push notifications to my REST end point through SNS and successfully applied the required business logic on those recipients.
Points:
You must be pushing multiple emails to that Recipient that's why more than one (unique) bounce notifications are being pushed by SNS.
Yes. And every notification has its own "messageId"
But why I am getting bounce Notifications for single recipient repeatedly, only, why not other wrong recipients.?
Response/s is attached:
{
"notificationType": "Bounce",
"bounce": {
"bounceType": "Permanent",
"bounceSubType": "Suppressed",
"bouncedRecipients": [
{
"emailAddress": "abc.xyz.aws#gmail.com",
"action": "failed",
"status": "5.1.1",
"diagnosticCode": "Amazon SES has suppressed sending to this ..."
}
],
"timestamp": "2017-12-28T07:06:37.106Z",
"feedbackId": "010101609bef62ff-0f30cac8-ec3c-41f3-9eee-5f7d90ef3c46-000000",
"reportingMTA": "dns; amazonses.com"
},
"mail": {
"timestamp": "2017-12-28T07:03:44.000Z",
"source": "abcXYZ#gmail.com",
"sourceArn": "arn:aws:ses:com-xyz-9:9696845301:identity/abcXYZ#gmail.com",
"sourceIp": "10.eee.ppq.lmn",
"sendingAccountId": "12365498512",
"messageId": "010101609bf1b35c-889e815e-b84c-4b08-a283-ee5f3e9740f2-000000",
"destination": ["abc.xyz.aws#gmail.com"]
}
}
Another one and all others:
{
"notificationType": "Bounce",
"bounce": {
"bounceType": "Permanent",
"bounceSubType": "Suppressed",
"bouncedRecipients": [
{
"emailAddress": "abc.xyz.aws#gmail.com",
"action": "failed",
"status": "5.1.1",
"diagnosticCode": "Amazon SES has suppressed sending to this ..."
}
],
"timestamp": "2017-12-28T07:06:37.106Z",
"feedbackId": "010101609befa8f5-a41662a5-b84e-4048-bc3a-d1f1469c8f1a-000000",
"reportingMTA": "dns; amazonses.com"
},
"mail": {
"timestamp": "2017-12-28T07:04:05.192Z",
"source": "abcXYZ#gmail.com",
"sourceArn": "arn:aws:ses:com-xyz-9:9696845301:identity/abcXYZ#gmail.com",
"sourceIp": "10.eee.ppq.lmn",
"sendingAccountId": "12365498512",
"messageId": "010101609bef61bb-310b4f29-0b01-4ecd-b763-b295e26d0732-000000",
"destination": ["abc.xyz.aws#gmail.com"]
}
}
For further info:
HTTP end point is is Spring MVC based REST controller.
Searching:
Through searching I found these:
1- Duplicate delivery/bounce notifications being sent to my SNS topic?
2- Emails being sent multiple times - can I see AWS logs ?
Kindly can any one help/guide or point out what can be the issue?
Hope you people are having a great holiday season.
"bounceSubType": "Suppressed",
This recipient is on the global suppression list, which means no SES customer can send mail to this address. It also means that these bounces are handled differently and more quickly than some other kinds of bounces.
https://aws.amazon.com/blogs/ses/goodbye-blacklist-introducing-the-suppression-list/
The bottom line here is that this bounce is a red flag to you that you need to manage your recipient addresses more carefully. Continuing to send messages to recipients on the suppression list is a signal to AWS that you may not be monitoring your bounces, or that you may not really be sending messages to people who actually want to receive them... and that in turn could potentially lead to enforcement actions.
If the emails you send result in excessive bounces, complaints, or other issues, your sending abilities might be placed on probation or suspended. This process is called enforcement.
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/e-faq.html

How to send Cloudwatch log details via email?

The diagram below is what I am trying to achieve. In brief, to send CloudTrail logs to CloudWatch log group then scan it for certain events and finally send email alerts if there is an concerting event.
I am following this official documentation which also has a sample CloudFormation templates: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-cloudformation-template-to-create-cloudwatch-alarms.html
Using the CloudFormation templates above, I have been able to send the email alerts. However the alerts are very basic; it does not send key information like which user initiated this event, when did it occur etc.
Logically thinking AWS::Logs::MetricFilter should pass the value to AWS::CloudWatch::Alarm which would then send the information. I have looked at the documentation of both MetricFilter and Alarm services. Dimension comes closer to what I want but not yet able to read the information from logs.
I would have thought this is a common use case and there would be documentation. Am I missing something glaringly obvious here? Has anyone here solved this issue?
AWS::Logs::MetricFilter block:
"AuthorizationFailuresMetricFilter": {
"Type": "AWS::Logs::MetricFilter",
"Properties": {
"LogGroupName": { "Ref" : "LogGroupName" },
"FilterPattern": "{ ($.errorCode = \"*UnauthorizedOperation\") || ($.errorCode = \"AccessDenied*\") }",
"MetricTransformations": [
{
"MetricNamespace": "CloudTrailMetrics",
"MetricName": "AuthorizationFailureCount",
"MetricValue": "1"
}
]
}
},
AWS::CloudWatch::Alarm block
"AuthorizationFailuresAlarm": {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"AlarmName" : "CloudTrailAuthorizationFailures",
"AlarmDescription" : "Alarms when an unauthorized API call is made.",
"AlarmActions" : [{ "Ref" : "AlarmNotificationTopic" }],
"Dimensions": [
{
"Name": "errorCode",
"Value": ""
},
{
"Name": "userIdentity",
"Value": ""
}
],
"MetricName" : "AuthorizationFailureCount",
"Namespace" : "CloudTrailMetrics",
"ComparisonOperator" : "GreaterThanOrEqualToThreshold",
"EvaluationPeriods" : "1",
"Period" : "300",
"Statistic" : "Sum",
"Threshold" : "1"
}
},
This is not possible.
Amazon CloudWatch Logs will accept information from AWS CloudTrail and, upon finding messages that match a pre-defined filter, will increment a metric count.
An Amazon CloudWatch alarm can then be triggered when the metric exceeds a certain threshold. However, there is no direct connection between the incoming data that generated the metrics and the alarm that triggers based upon the threshold.
Think of it like a turnstile counting people who enter a subway. The turnstile counts the number of people, but does not retain information about the people who passed through. In the same way, the CloudWatch alarm counts the events but does not have any information about the events that were counted.