I'm building an application that enables clients to book appointments with service providers. I'm using SNS -> SQS -> Lambda to process various emails that need to be sent when booking an appointment. IE I currently send an SNS message like so (in node.js):
await sns.publish({
Message: 'booking-request',
TopicArn: process.env.AWS_BOOKING_REQUEST_TOPIC_ARN,
MessageAttributes: {
artistEmail: SNSMF.string(artist.email),
artistName: SNSMF.string(artist.name),
clientEmail: SNSMF.string(req.body.email),
clientName: SNSMF.string(`${req.body.firstName} ${req.body.lastName}`),
date: SNSMF.string(moment(req.body.date).tz(studio.timeZone).format())
}
}).promise();
This all works fine, but I'm using MessageAttributes to pass the pertinent appointment details so my notifications layer can send the proper emails.
My main questions is, am I using MessageAttributes in the proper way, or is there a better way to pass all of this data? Should the data be the message itself? I ask because I believe that you can only have 10 MessageAttributes and I'm going to run into a limit with the appointment details (currently collecting about 10-12 data points about the appointment that I want to include in the emails). Any ideas? Thank you!
Normally, the 'main' information you wish to pass would be in the Body of the message. It is quite common the use JSON to pass various types of information.
MessageAttributes are normally something about the message itself rather than the content of the message, such as timestamps, priority and user information.
Given your requirements, I putting your data in the Body (eg in JSON) would avoid hitting limits and would also be more extensible.
Related
In AWS IOT, I'm able to publish an MQTT message from a device to the topic "topic/messages" :
{
"id":"messageID",
"value":"messageValue"
}
And I want to "augment" it in the server by adding a timestamp on it and let the message to continue on the SAME TOPIC.
So I'd like that subscribers on "topic/messages" receive this:
{
"id":"messageID",
"value":"messageValue",
"serverTimestamp":"1637867431920" <--- Here
}
However, I don't find the way how to process this message and let it flow in the same topic:
I can add a rule
SELECT * , timestamp() as serverTimestamp FROM 'topic/#'
But the rule does not augment the original message but creates an augmented copy and redirects it to some other service (Lambda, DynamoDB, Republish, etc..)
Those services work with the copy of the given value, but not with the original message, so the subscribers still receives the original sent message.
{
"id":"messageID",
"value":"messageValue"
}
I can republish the message using the same topic BUT as the topic has an attached rule, after the republishing, the rule's action is triggered again and again in a recursive loop)...
All the AWS examples I've read are meant to take the message, transform it , and do some other different thing with it (save in DynamoDB, save in a Bucket, end to Salesforce....) but none of them modify the message to be sent.
So what I'm looking for is a way to receive the message, add a field (or more) to it and let it flow in the same topic .
What is the simplest way to do this?
currently i have a cloud-build application which is being kicked off by a pub-sub trigger , subscribing to eg. topic1
I would like to know if i can kick off another cloud-build application from subscribing to the same topic. Is there a way to configure the message (or the trigger) so that if message1 is published to topic1, then cloudbuild1 is kicked off, and if message2 is published to topic1, then cloudbuild2 is kicked off?
Kind regards
marco
When you create a subscription on a topic, all the published messages in the topic are replicated in each subscription.
Therefore, if you have TOPIC and Sub1 and Sub2, if you publish 1 message in TOPIC, you will have this message in Sub1 and Sub2.
However, you can set up a filter on messages when you create a subscription. You can set this filter only at the creation and you can't update it later. You need to delete and recreate the subscription if you want to update the filter.
In addition, you can filter only on message attributes, not on the message body content.
Therefore, with filter, think wisely your filter from the beginning and when you publish a message in TOPIC, add attributes that allow your to route the messages to the correct subscription.
Currently, the message specified in the Document field while creating alerting policy appears in the Document field of the Stackdriver alert email.
I would like to overwrite the entire email message body with my custom content.
How can I overwrite the message body of Stackdriver Alert email with my custom message?
Is there any other workaround to do this?
You should be able to send the notification to a webhook, and this could directly be an HTTP-triggered Cloud Function.
This Cloud Function would receive all the information from the alert, and you can follow this tutorial to use SendGrid to send your alerts.
This is a lot more complex than just setting the email notifications, but also provides you with an amazing flexibility regarding alerts, as you'll be able to not just write the message however you want, but you could process the data in any way you want:
You have low priority alerts? Then store them and just send a digest
once in a while instead of spamming.
Want to change who is sent the
alert depending on a calendar rotation? Use the function to look up
who should be notified.
And those are just some random quick ideas I got while writing this message.
The information provided in the POST body is this one (that's just a sample):
{
"incident": {
"incident_id": "f2e08c333dc64cb09f75eaab355393bz",
"resource_id": "i-4a266a2d",
"resource_name": "webserver-85",
"state": "open",
"started_at": 1385085727,
"ended_at": null,
"policy_name": "Webserver Health",
"condition_name": "CPU usage",
"url": "https://app.google.stackdriver.com/incidents/f333dc64z",
"summary": "CPU for webserver-85 is above the threshold of 1% with a value of 28.5%"
},
"version": 1.1
}
You can create a single webhook that handles all the alerts, or you can create a webhook on a per-policy basis to handle things separately.
What is the purpose of using message body in SQS while you're already able to add message attributes?
Let's take an example, we want to push a message to new-user queue when a new user registered, I imagine the message will have an attribute userId, I don't see the use of body here.
Message attributes are supposed to be used as message metadata (like timestamp or possibly some category) and not the message itself.
Ideally, message payload should be given in the message body
So, for example if you are supporting JSON and XML payloads then possibly you can put payload type as message attribute and then when you fetch the message, based on this payload type attribute you decide between the JSON message processor or XML message processor. This is just a superficial example to explain the usage of attributes and body
Following is the extract from AWS Doc
Amazon SQS provides support for message attributes. Message attributes allow you to provide structured metadata items (such as timestamps, geospatial data, signatures, and identifiers) about the message. Message attributes are optional and separate from, but sent along with, the message body. This information can be used by the receiver of the message to help decide how to handle the message without having to first process the message body. Each message can have up to 10 attributes. To specify message attributes, you can use the AWS Management Console, AWS software development kits (SDKs), or query API.
To map with the traditional queue provider such as rabbitMQ or Kafka world.
We can understand as below:
message_body=message_payload
message_attributes=message_headers ( can be used to apply different routing and filtering message using their headers information)
In fact, I prefer the term payload and headers more than what terms used in aws sqs, its abit confusing.
Message attributes sound more like the attributes of message payload
I'm trying to follow along the amazon sns publish example using the Amazon documentation site but it's vague on Message, MessageAttributes and MessageStructure.
First of all, is the Message property going to be a string even if you set MessageStructure to json? e.g. If I want to send an object instead of just a string message. If it's string - do I need to JSON.stringify my object before passing it as a Message property?
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html#publish-property
Should I be doing this using MessageAttributes instead? What is that property - the amazon documentation merely states it "Message attributes for Publish action" which seems like a tautology.
http://docs.aws.amazon.com/sns/latest/APIReference/API_Publish.html
Setting MessageStructure to json is only used if you are going to send a json-formatted message structure to SNS in the specific jormat SNS understands. This is only used when you are publishing to multiple endpoint types and want to vary the message body by endpoint type. This isn't the same as "I want to send a message where the body has been serialized as JSON."
If you are sending "a JSON object," you need to stringify it, and send it just as you would any other (non-JSON) messages, because SNS messages are, fundamentally, strings.
MessageAttributes are something else entirely. They allow you to send pseudo-out-of-band key/value pairs along with your message, which can be useful for example if your message has been gzipped and base64 encoded (again, for example) you could attach an "external_id" attribute that the recipient could evaluate to decide whether it needed to unpack the whole message or could just discard it.