Qpid offers responses to the received message. For parallel processing purposes we have multiple instances of the application that are listening to the same queue. The problem is that I would like that exactly the same instance receives the response. I believe that this can be done using correlation ids, but I cannot find an example how to do that with Qpid.
I searched the web and found discussion from 2009 (http://qpid.2158936.n2.nabble.com/get-message-by-correlationId-td3440876.html) where they aimed to add selectors shortly. This is probably the associated ticket (https://issues.apache.org/jira/browse/QPID-4558). There is finished subtask "Initial selector implementation", so I belive that this is implemented.
I was searching the API documentation and found methods getCorrelationId() and setCorrelationId(), but I cannot find an example how to filter the messages from queue.
Thanks.
Related
I've just successfully implemented a JMS-message-processor in my WSO2-process. However, when logged in as an admin on the ActiveMQ console, I can view the stats of the queue but I cannot access the contents of the pending message. Instead, I see this error:
Cannot display ObjectMessage body. Reason: Failed to build body from content. Serializable class not available to broker. Reason: java.lang.ClassNotFoundException: org.apache.synapse.message.store.impl.commons.StorableMessage
Accessing the contents of a pending message is possible when using the RabbitMQ, so I figured it should also be possible with this message broker.
I've found solutions online, but they don't seem to be working, or perhaps they're out of date. Can somebody point me to the correct answer to solve this? I'm on WSO2 Integration Studio 8.1.0 and am using ActiveMQ 5.17.1.
It appears that WSO2 is using javax.jms.ObjectMessage despite the fact that there's a long list of reasons why doing so is a bad idea, including the issue you're currently facing, but I digress.
The problem here is that the implementation of the javax.jms.ObjectMessage (i.e. org.apache.synapse.message.store.impl.commons.StorableMessage in this case) must be on the classpath of any application which wants to deserialize that message. This class is not, in fact, on the classpath of ActiveMQ therefore the ActiveMQ web console cannot deserialize the message and display its contents (assuming those contents are human-readable in the first place). That's why a ClassNotFoundException is thrown.
You may be able to resolve the issue by putting org.apache.synapse.message.store.impl.commons.StorableMessage on ActiveMQ's classpath. Aside from that there's really nothing to be done.
I assume this is different for RabbitMQ because in that case WSO2 doesn't use the JMS API and uses the AMQP protocol which is not Java-centric like JMS.
Justin has explained the cause of the issue and as suggested you can try adding the class to ActiveMQ runtime and see whether it resolves the issue. This class is located in the Synapse-Core(synapse-core_2.x.x.wso2vXXX.jar). But remember although this may resolve the ActiveMQ UI issue, this message is not consumable by other systems unless they know how to deserialize it.
Let me add more details on why the message is serialized. Message serialization happens when you use the Store mediator, and when you store a message with Store Mediator it is intended to be only read by a Message Processor. Simply the serialized message can only be consumed by WSO2 Message Processors. In other words, Store Mediator and the Message processors are tightly coupled.
If message serialization is an issue for you. For example, if other systems are consuming the messages that WSO2 publishes you can try using the JMS transport to produce and to consume messages from ActiveMQ. Other than that you can also consider using JMS Inbound Endpoint to consume messages, which all use standard media types when storing the message.
I am looking into ways to order list of messages from google cloud pub/sub. The documentation says:
Have a way to determine from all messages it has currently received whether or not there are messages it has not yet received that it needs to process first.
...is possible by using Cloud Monitoring to keep track of the pubsub.googleapis.com/subscription/oldest_unacked_message_age metric. A subscriber would temporarily put all messages in some persistent storage and ack the messages. It would periodically check the oldest unacked message age and check against the publish timestamps of the messages in storage. All messages published before the oldest unacked message are guaranteed to have been received, so those messages can be removed from persistent storage and processed in order.
I tested it locally and this approach seems to be working fine.
I have one gripe with it however, and this is not something easily testable by myself.
This solution relies on server-side assigned (by google) publish_time attribute. How does Google avoid the issues of skewed clocks?
If my producer publishes messages A and then immediately B, how can I be sure that A.publish_time < B.publish_time is true? Especially considering that the same documentation page mentions internal load-balancers in the architecture of the solution. Is Google Pub/Sub using atomic clocks to synchronize time on the very first machines which see messages and enrich those messages with the current time?
There is an implicit assumption in the recommended solution that the clocks on all the servers are synchronized. But the documentation never explains if that is true or how it is achieved so I feel a bit uneasy about the solution. Does it work under very high load?
Notice I am only interested in relative order of confirmed messages published after each other. If two messages are published simultaneously, I don't care about the order of them between each other. It can be A, B or B, A. I only want to make sure that if B is published after A is published, then I can sort them in that order on retrieval.
Is the aforementioned solution only "best-effort" or are there actual guarantees about this behavior?
There are two sides to ordered message delivery: establishing an order of messages on the publish side and having an established order of processing messages on the subscribe side. The document to which you refer is mostly concerned with the latter, particularly when it comes to using oldest_unacked_message_age. When using this method, one can know that if message A has a publish timestamp that is less than the publish timestamp for message B, then a subscriber will always process message A before processing message B. Essentially, once the order is established (via publish timestamps), it will be consistent. This works if it is okay for the Cloud Pub/Sub service itself to establish the ordering of messages.
Publish timestamps are not synchronized across servers and so if it is necessary for the order to be established by the publishers, it will be necessary for the publishers to provide a timestamp (or sequence number) as an attribute that is used for ordering in the subscriber (and synchronized across publishers). The subscriber would sort message by this user-provided timestamp instead of by the publish timestamp. The oldest_unacked_message_age will no longer be exact because it is tied to the publish timestamp. One could be more conservative and only consider messages ordered that are older than oldest_unacked_message_age minus some delta to account for this discrepancy.
Google Cloud Pub-sub does not guarantee order of events receive to consumers as they were produced. Reason behind that is Google Cloud Pub-sub also running on a cluster of nodes. The possibility is there an event B can reach the consumer before event A. To Ensure ordering you have to make changes on both producer and consumer to identify the order of events. Here is section from docs.
In the context of writing a Messenger chat bot in a cloud environment, I'm facing some concurrency issues.
Specifically, I would like to ensure that incoming messages from the same conversation are processed one after the other.
As a constraint, I'm processing the messages with workers in a Cloud environment (i.e the worker pool is of variable size and worker instances are potentially short-lived and may crash). Also, low latency is important.
So abstracting a little, my requirements are:
I have a stream of incoming messages
each of these messages has a 'topic key' (the conversation id)
the set of topics is not known ahead-of-time and is virtually infinite
I want to ensure that messages of the same topic are processed serially
on a cluster of potentially ephemeral workers
if possible, I would like reliability guarantees e.g making sure that each message is processed exactly once.
My questions are:
Is there a name for this concurrency scenario?.
Are there technologies (message brokers, coordination services, etc.) which implement this out of the box?
If not, what algorithms can I use to implement this on top of lower-level concurrency tools? (distributed locks, actors, queues, etc.)
I don't know of a widely-accepted name for the scenario, but a common strategy to solve that type of problem is to route your messages so that all messages with the same topic key end up at the same destination. A couple of technologies that will do this for you:
With Apache ActiveMQ, HornetQ, or Apache ActiveMQ Artemis, you could use your topic key as the JMSXGroupId to ensure all messages with the same topic key are processed in-order by the same consumer, with failover
With Apache Kafka, you could use your topic key as the partition key, which will also ensure all messages with the same topic key are processed in-order by the same consumer
Some message broker vendors refer to this requirement as Message Grouping, Sticky Sessions, or Sticky Message Load Balancing.
Another common strategy on messaging systems with weaker delivery/ordering guarantees (like Amazon SQS) is to simply include a sequence number in the message and leave it up to the destination to resequence and request redelivery of missing messages as needed.
I think you can fix this by using a queue and a set. What I can think of is sending every message object in queue and processing it as first in first out. But while adding it in queue add topic name in set and while taking it out for processing remove topic name from set.
So now if you have any topic in set then don't add another message object of same topic in queue.
I hope this will help you. All the best :)
We have a BizTalk application which sends XML files to external applications by using a web-service.
BizTalk calls the web-services method by passing XML file and destination application URL as parameters.
If the external applications are not able to receive the XML, or if there is no response received from the web-service back to BizTalk the message gets suspended in BizTalk.
Presently for this situation we manually go to BizTalk admin and resume each suspended message.
Our clients want this process to be automated all, they want an dashboard which shows list of message details and a button, on its click all the suspended messages have to be resumed.
If you are doing this within an orchestration and catching the connection error, just add a delay shape configured to 5 hours. Or set a retry interval to 300 minutes and multiple retries on the send port if that makes sense. You can do this using the rule engine as well.
Why not implement an asynchronous pattern?
You make it so, so that the orchestration sends the file out via a send shape while initializing a certain correlation set.
You then put a listen shape with at one end:
- the receive (following the initialized correlation set)
- a delay shape set to 5 hours.
When you receive the message, your orchestration can handle it gracefully.
When you don't, the delay shape will kick in and you handle accordingly.
Benefit to this solution in comparison to the solution of 40Alpha will be that your orchestration will only 'wake up' from a dehydrated state if the timeout kicks in OR when the response is received. In the example of 40Alpha, the orchestration would wake up a lot of times, consuming extra resources.
You may want to look a product like BizTalk 360. It has those sort of monitoring and command built into it. I'm not sure it works with BizTalk 2006R2 though, but you should be thinking about moving off that platform anyway as it is going out of Microsoft support.
I already posted a question here, and I finally found the source of the problem myself. But the issue remains unresolved.
Arriving messages are stored in a queue until a matching activity is found. But if the activities are not activated in the same order as the messages arrived, the process will block.
I modelled a diagram which is available here, the corresponding BPEL package can be downloaded from here.
The first process remains blocked on Receive2 whereas the second completes.
I'm testing on the BPS version 2.1.2
When multiple processes are communicating with each other, I do not always know if a message will arrive before or after another one. I think this should be possible, I see no reason why not: the messages are present in the H2 database, so a suitable query sould be able to return them.
Thanks in advance for your help.