What is the purpose of using message body in SQS while you're already able to add message attributes?
Let's take an example, we want to push a message to new-user queue when a new user registered, I imagine the message will have an attribute userId, I don't see the use of body here.
Message attributes are supposed to be used as message metadata (like timestamp or possibly some category) and not the message itself.
Ideally, message payload should be given in the message body
So, for example if you are supporting JSON and XML payloads then possibly you can put payload type as message attribute and then when you fetch the message, based on this payload type attribute you decide between the JSON message processor or XML message processor. This is just a superficial example to explain the usage of attributes and body
Following is the extract from AWS Doc
Amazon SQS provides support for message attributes. Message attributes allow you to provide structured metadata items (such as timestamps, geospatial data, signatures, and identifiers) about the message. Message attributes are optional and separate from, but sent along with, the message body. This information can be used by the receiver of the message to help decide how to handle the message without having to first process the message body. Each message can have up to 10 attributes. To specify message attributes, you can use the AWS Management Console, AWS software development kits (SDKs), or query API.
To map with the traditional queue provider such as rabbitMQ or Kafka world.
We can understand as below:
message_body=message_payload
message_attributes=message_headers ( can be used to apply different routing and filtering message using their headers information)
In fact, I prefer the term payload and headers more than what terms used in aws sqs, its abit confusing.
Message attributes sound more like the attributes of message payload
Related
Problem: My use case is I want to publish thousends of messages to Google Cloud Pub/Sub with a 5min retention period but only retrieve specific messages by their ID - So a cloud function will retrieve one message by ID using the Nodejs SDK and all the untreated messages will be deleted by the retention policy. All the current examples mention are to handle random messages from the subscriber.
Is it possible to just pull 1 message by id or any other metadata and close the connection.
There is no way to retrieve individual messages by ID, no. It doesn't really fit into the expected use cases for Cloud Pub/Sub where the publishers and subscribers are meant to be decoupled, meaning the subscriber inherently doesn't know the message IDs prior to receiving the messages.
You may instead want to transmit the messages via whatever mechanism you are using to making the subscribers aware of the message IDs. Or, if you know at publish time which messages will ultimately need to be retrieved, you could add an attribute to the message to indicate this and use filtering.
I'd like to use AWS S3 to allow some users to add files to an S3 bucket.
Then, I'd like to generate an SQS message when a new file has been added.
Last, I'd like to consume the SQS message and process it with a background job worker of a particular class.
I'd like to use SQS message attributes to determine which background job worker class I should use for processing. As the SQS message attribute documentation states:
Message attributes [...] can be used by the consumer of the message to help decide how to handle the message without having to first process the message body.
(via the SQS Developer Guide)
However, under the S3 bucket's Properties, under Advanced Settings, the Events settings do not appear to expose a way to set message attributes.
Is there a way to specify message attributes on an event-by-event basis for events generated from S3?
There is not a way to inject custom message attributes into S3 event notifications... but also note that you may be misinterpreting what message attributes can be used for.
Message attributes [...] can be used by the consumer of the message
This means they provide a mechanism for the consumer to triage a message after the consumer has already received it from the queue.
You can't selectively consume messages based on message attributes. Queue consumers receive the next available message(s) when they poll the queue. They don't get to select which messages they consume.
If you want to divide the messages up by class, you'll need an intermediate process that selectively distributes messages to the appropriate (separate) downstream queues. Better, if your structure allows it, might be different event configurations matching specific patterns that need to go to individual queues.
You should use a Lambda to add messages to the queue, for example with python using BOTO 3 API. Map this Lambda to the S3 Event. http://docs.aws.amazon.com/lambda/latest/dg/with-s3.html
I want to distinguish the messages after retreiving from aws fifo sqs for routing to correct processor. And I planned to utilize the MessageGroupId , which I set while sending the request.
I am using Java sdk.
sendMessageRequest.setMessageGroupId(messageGroupId);
Documentation says it can be retrieved as request parameter of AttributeName
MessageGroupId - Returns the value provided by the sender that calls the SendMessage action. Messages with the same MessageGroupId are returned in sequence.
However, I could not find the correct way/method to do so. There is no attribute
MessageGroupId present and size is 0.
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages)
{
System.out.println(" Size getAttributes: " + message.getAttributes() .size());
}
Only if I set any attribute explicitly then it is retreivable (like with MESSAGEPRIORITY), but how to receive the values of attributes like MessageGroupId?
Help is appreciated.
You have to tell SQS which attributes you want it to give you, before getAttributes() will be able to show them to you.
List<Message> messages = sqs.receiveMessage(receiveMessageRequest.withAttributeNames("All")).getMessages();
Note that SQS messages have two different kinds of attributes. Attributes are system-generated (like MessageGroupId), while Message Attributes are user-generated, custom key/value pairs included when the message is sent but transported external to the message body itself.
Aws SQS giving different messages for the same parameters
So I posted a message using:
https://sqs.us-west-2.amazonaws.com/Otherinfo/?Action=SendMessage&MessageBody=Ola&MessageAttribute.1.Name=test1&MessageAttribute.1.Value.StringValue=Drizzy&MessageAttribute.1.Value.DataType=String
The important thing here is I want to get the messages based on their attributes. Then using postman I call for Receive:
https://sqs.us-west-2.amazonaws.com/Otherinfo/?Action=ReceiveMessage&MessageAttributeName.1=test1.*&WaitTimeSeconds=10
I even made it wait for response but then I keep getting different messages from the queue sometimes even when I make a mistake and enter a wrong attribute name I still get a response. I'm I doing something wrong here?
Is there any other messaging system like this that would work with AngularJS?
If I understand your question, it sounds like you are trying to filter the messages received based on your attribute parameters - that is not how they work.
The attribute parameters is a way of specifying which attributes of the selected records should be returned - it does not apply a filter to the messages and only return the messages that match.
You can't filter messages received - you ask for messages, SQS sends you messages - you can't control the order that you get them, or the selection criteria.
I'm trying to follow along the amazon sns publish example using the Amazon documentation site but it's vague on Message, MessageAttributes and MessageStructure.
First of all, is the Message property going to be a string even if you set MessageStructure to json? e.g. If I want to send an object instead of just a string message. If it's string - do I need to JSON.stringify my object before passing it as a Message property?
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html#publish-property
Should I be doing this using MessageAttributes instead? What is that property - the amazon documentation merely states it "Message attributes for Publish action" which seems like a tautology.
http://docs.aws.amazon.com/sns/latest/APIReference/API_Publish.html
Setting MessageStructure to json is only used if you are going to send a json-formatted message structure to SNS in the specific jormat SNS understands. This is only used when you are publishing to multiple endpoint types and want to vary the message body by endpoint type. This isn't the same as "I want to send a message where the body has been serialized as JSON."
If you are sending "a JSON object," you need to stringify it, and send it just as you would any other (non-JSON) messages, because SNS messages are, fundamentally, strings.
MessageAttributes are something else entirely. They allow you to send pseudo-out-of-band key/value pairs along with your message, which can be useful for example if your message has been gzipped and base64 encoded (again, for example) you could attach an "external_id" attribute that the recipient could evaluate to decide whether it needed to unpack the whole message or could just discard it.