Aws SQS giving different messages for the same parameters
So I posted a message using:
https://sqs.us-west-2.amazonaws.com/Otherinfo/?Action=SendMessage&MessageBody=Ola&MessageAttribute.1.Name=test1&MessageAttribute.1.Value.StringValue=Drizzy&MessageAttribute.1.Value.DataType=String
The important thing here is I want to get the messages based on their attributes. Then using postman I call for Receive:
https://sqs.us-west-2.amazonaws.com/Otherinfo/?Action=ReceiveMessage&MessageAttributeName.1=test1.*&WaitTimeSeconds=10
I even made it wait for response but then I keep getting different messages from the queue sometimes even when I make a mistake and enter a wrong attribute name I still get a response. I'm I doing something wrong here?
Is there any other messaging system like this that would work with AngularJS?
If I understand your question, it sounds like you are trying to filter the messages received based on your attribute parameters - that is not how they work.
The attribute parameters is a way of specifying which attributes of the selected records should be returned - it does not apply a filter to the messages and only return the messages that match.
You can't filter messages received - you ask for messages, SQS sends you messages - you can't control the order that you get them, or the selection criteria.
Related
Problem: My use case is I want to publish thousends of messages to Google Cloud Pub/Sub with a 5min retention period but only retrieve specific messages by their ID - So a cloud function will retrieve one message by ID using the Nodejs SDK and all the untreated messages will be deleted by the retention policy. All the current examples mention are to handle random messages from the subscriber.
Is it possible to just pull 1 message by id or any other metadata and close the connection.
There is no way to retrieve individual messages by ID, no. It doesn't really fit into the expected use cases for Cloud Pub/Sub where the publishers and subscribers are meant to be decoupled, meaning the subscriber inherently doesn't know the message IDs prior to receiving the messages.
You may instead want to transmit the messages via whatever mechanism you are using to making the subscribers aware of the message IDs. Or, if you know at publish time which messages will ultimately need to be retrieved, you could add an attribute to the message to indicate this and use filtering.
So I am building a small application that uses SQS. I have a simple handler process that determines if a given message is considered processed, marked for retry (to be re-queued) or is not able to be processed (should be sent to dead letter).
However based on the docs it would appear the only way to truly send a message to DL is by using a redrive policy which operates over # of receives a message has racked up. Because of the nature of my application, I could have several valid retries if my process isn't ready to handle a given message, but there are also times I may want to DL a message I have just received. Does AWS/Boto3 not provide a way to mark a specific message for DL?
I know I can just send the message myself to another queue I consider my own DL, I would just rather use AWS' built in tools for this.
I don't believe there is any limitation that would prevent you from sending the message to the deal-letter-queue by yourself.
So just read the message from the Q, if you know it needs to go to the DLQ directly, send it to the DLQ and remove it from the regular Q.
I am using AWS SQS and Spring JMS in my project. I have my method with #JmsListener(destination = "queue_name"). I want to use this queue for two different types of messages.
Since this listener is configured to this queue it receives both types of messages. What I am trying to achieve is to ignore message of one type. (Sender is adding a MessageAttribute while sending message to Queue). So, is there a way to just ignore message coming from sender 2 so this method won't process them.
Also, I have DLQ set on this queue with max receives as 5. So if message is not processed in first 5 attempts it gets moved to DLQ.
Please do share your suggestion.
Thanks.
The correct solution is to use 2 different queues; SQS can't filter the messages delivered by any property, so as you are seeing, when the client reads the message and doesn't process it, it is going to end up in your DLQ quicker.
Queues are free, so having multiple won't cost any more.
What is the purpose of using message body in SQS while you're already able to add message attributes?
Let's take an example, we want to push a message to new-user queue when a new user registered, I imagine the message will have an attribute userId, I don't see the use of body here.
Message attributes are supposed to be used as message metadata (like timestamp or possibly some category) and not the message itself.
Ideally, message payload should be given in the message body
So, for example if you are supporting JSON and XML payloads then possibly you can put payload type as message attribute and then when you fetch the message, based on this payload type attribute you decide between the JSON message processor or XML message processor. This is just a superficial example to explain the usage of attributes and body
Following is the extract from AWS Doc
Amazon SQS provides support for message attributes. Message attributes allow you to provide structured metadata items (such as timestamps, geospatial data, signatures, and identifiers) about the message. Message attributes are optional and separate from, but sent along with, the message body. This information can be used by the receiver of the message to help decide how to handle the message without having to first process the message body. Each message can have up to 10 attributes. To specify message attributes, you can use the AWS Management Console, AWS software development kits (SDKs), or query API.
To map with the traditional queue provider such as rabbitMQ or Kafka world.
We can understand as below:
message_body=message_payload
message_attributes=message_headers ( can be used to apply different routing and filtering message using their headers information)
In fact, I prefer the term payload and headers more than what terms used in aws sqs, its abit confusing.
Message attributes sound more like the attributes of message payload
I have a SQS producer and lots of consumers. It would be very helpful to know if the producer can tell if a particular message has been deleted by a consumer or not. Is there a way of doing this? I'm currently using boto 2.6.0.
As far as I know, SQS does not provide any mechanism to be notified when a message is deleted. So, I think if you want to know when messages are deleted, you will have to keep track of that separately by keeping a database of message ids and having consumers tell you the message id of any messages they have deleted.
As far as I know you can't track an message in a Queue. Depending on your goal you could try the following things:
Monitoring
Write the results of a job to a logfile, and maybe use something like logstash with Kibana. If you get creative you might even fire things straight into something like ElasticSearch or SimpleDB.
Callback
The receivers could fire any kind of "callback" to the processor or any other process updating a certain message state in for example a database or a cache.
You have to keep in mind that this means that while you scale up your receivers, your processor has to scale up as well. Also keep an eye on your indexes, make sure your status update "write" is fast.