Problem:
Process a backlog of messages where each message has three headers "service", "client", and "stream". I want to process the backlog of messages with maximum concurrency, but I have some requirements:
Only 10 messages with the same service can be processing at once.
Only 4 messages with the same service AND client can be processing at
once.
All messages with the same service AND client AND stream must
be kept in order.
Additional Information:
I've been playing around with "maxConcurrentConsumers" along with the "JMSXGroupID" in a ServiceMix (Camel + ActiveMQ) context, and I seem to be able to get 2 out of 3 of my requirements satisfied.
For example, if I do some content-based routing to split the backlog up into separate "service" queues (one queue for each service), then I can set the JMSXGroupID to (service + client + stream), and set maxConcurrentConsumers=10 on routes consuming from each queue. This solves the first and last requirements, but I may have too many messages for the same client processing at the same time.
Please note that if a solution requires a separate queue and route for every single combination of service+client, that would become unmanageable because there could be 10s of thousands of combinations.
Any feedback is greatly appreciated! If my question is unclear, please feel free to suggest how I can improve it.
To my knowledge, this would be very hard to achieve if you have 10k+ combos.
You can get around one queue per service/client combo by using consumers and selectors. That would, however, be almost equally hard to deal with (you simply don't create 10k+ selector consumers unharmed and without significant performance considerations), if you cannot predict in some way a limited set of service/client active at once.
Can you elaborate on the second requirement? Do you need it to make sure there are some sense of fairness among your clients? Please elaborate and I'll update if I can think of anything else.
Update:
Instead of consuming by just listening to messages, you could possibly do a browse on the queue, looping through the messages and pick one that "has free slots". You can probably figure out if the limit has been reached by some shared variable that keeps track given you run in a single instance.
Related
We have adopted an AWS powered microservice architecture where different sorts of payloads enter the system with a UUID and type via mysql.lambda_async from our database.
The problem is, that we've noticed that messages can come out of order. Imagine the scenario with the following type of message:
DEASSIGN_ROLE
ASSIGN_ROLE
When the actual intention was a quick toggle:
ASSIGN_ROLE
DEASSIGN_ROLE
Now we have a user with the wrong (elevated) permissions.
I've done some cursory research and for example answers like Handling out of order events in CQRS read side suggest using sequence numbers.
Introducing a sequence number would be quite hard as we have many different types of messages. A sequence number would require a syncronous counter, where we have gone great pains to be simply asynchronous. Bear in mind, our system that generates the message is an SQL trigger ultimately.
Are there simpler solutions I am missing?
I would say there is an unsolvable problem :
you want to be full asynchronous
you need sequentiality in your results
We had the same problem as yours, and we ended by setting sequences by type of messages.
Trying to be asynchronous and parallel when possible (according to message types/topics)
I have the task of scheduling & executing a lot of web-request, in java 8, with the following conditions:
each web-request belongs to exactly one of distinct groups
the group a web-request belongs to is an immutable and deterministic property of the request (i.e. is not a result of some (pseudo-)random logic), e.g. imagine the user on behalf the request is being made
the web-service is tracking the quota usage of web-requests it receives, for each of these distinct group
a web-request may receive an HTTP 429 (Too Many Requests) at any given moment, indicating that the quota for that group is full
when this happens, no more web-requests of the same group are allowed to be made until the stated time indicated by the Retry-After header of the response, such futile requests still count as part of the quota
web-requests of a non-throttled group can, and should be processed regardless of some other groups being throttled
some requests are more equal then others, therefore eligible requests should be processed in some priority order
the number of these distinct quota groups is in the few hundreds (for now)
at any given moment a new group may be born, e.g. a new user joins the organization
I've been collecting some ideas, none of which I am satisfied with:
The most obvious is that each group could be handled by their very own ThreadPoolExecutor consuming from a respective PriorityBlockingQueue
Simplicity has its virtues, but I kind of dislike running hundred instances of ThreadPoolExecutors (even if each and every one of them is using a single thread for execution).
I could (try to) go down the tedious and quite error-prone path of implementing my own BlockingQueue, with a PriorityQueue maintained for each group
the number of interface methods in BlockingQueue is not that many, but the designers of that concurrency library saw fit to extend Collection and Queue interfaces, and just the amount of time to implement all those methods and test them too sounds like a (dangerous) waste of time to me
I could also go and relax the goal of letting non-throttled groups&requests to progress, and just block all requests until stated time
this may not be as bad as it sounds, I will still have to check how easy it is to hit the quota limit and what the time penalty is - 5 minutes blackout ever other week sounds almost acceptable, half an hour every midnight is definitely not OK
Another idea is to have ThreadPoolExecutor with a single PriorityBlockingQueue (PBQ) with a map of throttled group -> request-lists on the side
on several occasions (on submit, on consuming from main PBQ, and even on just having received an HTTP 429 response), the group of the request would be tested for being throttled, and if it's that the case, the request would be put to that throttled group -> request list map
but normally, requests would just be consumed by the ThreadPoolExecutor
of course, whenever some throttling period indicated by Retry-After header of the HTTP 429 response has ended, the respective group would wake up and all it's requests would be re-submitted to the main PBQ
I've also been reading up on RxJava, but none of the delay, throttle or backpressure facilities are suitable - or at least I coulnd't see how.
At this point I really don't expect any pseudo-code (well, unless it's actually shorter that way), but I am most interested in better ideas or perhaps pointers to existing facilities.
(Btw, the web-service is the Microsoft Graph API, in case anyone wonders.)
I'm writing a specialized distributed storage system using akka clustering and would like to send large payloads (>1MB byte arrays) between actors. I found that I had to edit akka.remote.netty.tcp.maximum-frame-size to enable this.
My question is: are there any other performance implications that I need to take into account for this? For example, do I need to further tune netty buffer sizes? Is there a way to minimize the number of copies created?
One thing to think of is that large packages can cause something like head of line blocking where the system messages such as heartbeats will be blocked by sending the large package. In general it is a good idea to split large internode messages up into smaller messages to avoid this.
We are currently working on a new remoting subsystem for Akka which contains a separate large messages channel between nodes and where we also have a separation of regular messages and system messages to avoid those interfering with each other.
It is ubiquitously mentioned that - 'Celery is an asynchronous task queue/job queue based on distributed message passing'.Though I know how to use Celery workers and all.But deep down I dont understand the real importance and meaning of distributed messaging passing and role of task queue in it.
I have surfed much of the internet but nowhere is the clear explanation of the definition word by word.It is just always mentioned as a fact.
Can someone please here spare his/her time to explain all the importance of the terms with some relevant examples?
Sorry if this question looks trivial to most of the people but for me solution to this will go long way in understanding how things work.
Thanks.
To put it very simply "distributed" means that the work is distributed among many workers.
distribute
dɪˈstrɪbjuːt,ˈdɪstrɪbjuːt/Submit
verb
past tense: distributed; past participle: distributed
1.
give a share or a unit of (something) to each of a number of recipients.
when you run a task celery puts it on a queue, messages are passed to the workers, one of them runs the task.
I often heard this question coming from different sources, but never got a good idea of the technologies to achieve this. Can anyone shed some lights? The question is: you have a website which has high volume of users access per day. Your website is deployed in a distributed manner, have multiple webservers and load balancers responding incoming requests from lots of locations. How do you get the 1000000th user access, and show him a special page saying "congrats, you are our 1000000th visitor!". Assuming you had a distributed backend.
You could do it with jQuery, for example:
$("#linkOfInterest").click(function() { //code for updating a variable/record that contains the current number of clicks });
CSS:
a#linkOfInterest {
//style goes here
}
somewhere in the html :
<a id="linkOfInterest" href="somepage.htm"></a>
You are going to have to trade off performance or accuracy. The simplest way to do this would be have a memcached instance keep track of your visitor counts, or some other datastore with an atomic increment operation. Since there is only a single source of truth, only 1 visitor will get the message. This will delay the loading of your page by the roundtrip to the store at minimum.
If you can't afford the delay, then you will have to trade off accuracy. A distributed data store will not be able to atomically increment the field any faster than a single instance. Every web server can read and write to a local node, but another node at another datacenter may also reach 1 million users counts before the transactions are reconciled. In that case 2 or more people may get the 1 millionth user message.
It is possible to do so after the fact. Eventually, the data store will reconcile the increments, and your application can decide on a strict ordering. However, if you have already decided that a single atomic request takes too long, then this logic will take place too late to render your page.