I want to send a reply after I have persisted and updated the state of the actor using persistAll. Unfortunately I have not found a callback or onSucces handler to send back a reply after the last event has been persisted.
This is a shortcoming of the API, there is no built in way to react on all persistAll completing, you will have to keep a counter or a set of completed persists yourself and only trigger your logic when the last persist completes.
As far as I remember this cannot be easily fixed because it would break binary and source compatibility.
In the "next generation" persistent actors (in Akka typed) this works more as you would expect and the side effect you want to execute on successful persist of the events will only execute once, when all the events are complete.
Related
I am reading data and processing it further. if processing fails, I will not call checkpoint function. I hope that not checkpointing will stop further processing of events until issue is fixed. Is checkpointing sufficient for resiliency or I need to implement something like dead blob processor to provide failure handling?
Is checkpointing sufficient for resiliency
No not really. The processing will continue until the process hosting the processing logic stops. Say for example you have an azure funtion processing the message this will go on.
What does happen that in the event of a function restart (or whatever process you use to handle event hub messages) the function will start processing messages from the moment of the last checkpoint. Probably not what you want because the messages that caused a failure in the past will be processed again and probably fail again.
There is no dead lettering or retry mechanism out of the box, you will need to define that logic yourself.
So, TL;DR: the checkpoint its only purpose is to tell the processing logic where to start processing message from the backlog.
I have an actor processing messages and storing its results via asynchronous API (ReactiveMongo). IE when computation is completed actor is asking ReactiveMongo to store computation result and that call is non blocking.
How can I stop actor processing next messages until last ReactiveMongo request feature will be completed? Also mailbox should be able to receive incoming messages.
Blocking solution
Simple and wrong answer: you can do this by blocking the actor, just call Await (or whatever similar method in the language do you use).
It is wrong because Do not block inside the actor.
Not blocking solution
Master\Worker pattern is a good for this problem: http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2
So your worker actor will send the "Work Done" message after ReactiveMongo request feature completion. Then master actor will send new "Do this work" message to the worker.
I have an Akka based application that performs a large amount of work and reports progress by message passing. The "workers" report when they complete some work up to their "manager" who then feeds them more work and at the same time the "manager" sends a message (fire and forget) to an actor ref which simply keeps track of progress.
The "manager" and "workers" all share a dispatch thread pool, and the "tracker" actor sits in it's own small thread pool. I also throttle how much stuff gets posted to it using a basic modulo of the progress, so i only send a message when 20 items of work are completed each time. This seems to work ok for small numbers of jobs - say a couple of hundred. I see the tracker actor receiving the messages and updating its instance variable and responding to requests to retrieve the progress data.
However, when handling thousands of items of work, the progress count goes up for a little while and then appears to totally stop - as if no messages are being picked up by the tracker actor and they are all just filling up it's message queue. Is the "worker" dispatcher hogging all the resources perhaps? I thought that each dispatcher was supposed to be able to deliver messages to actors with some regularity. I've tried messing about with the "fairness" configuration in each dispatcher but it seems to make no difference.
I value anyone's thoughts on this.
I ended up changing some things around and putting the "manager" into a separate dispatcher and things started moving again. I ran 109,000 jobs through it no problem.
I am using Akka.NET to implement an actor system in which some actors are created on demand and are deleted after a configurable idle period (I use Akka's "ReceiveTimeout" mechanism for this). Each of these actors is identified by a key, and there should not exist two actors with the same key.
These actors are currently created and deleted by a common supervisor. The supervisor can be asked to return a reference to the actor matching a given key, either by returning an existing one or creating a new one, if an actor with this key doesn't exist yet. When an actor receives the "ReceiveTimeout" message, it notifies the supervisor who in turn kills it with a "PoisonPill".
I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception. Worse, when sending an "Ask" message, the sender remains blocked, waiting indefinitely (or until a timeout) for a response that he will never receive.
I first thought about Akka's "Deatchwatch" mechanism to monitor an actor's lifecycle. But, if I'm not mistaken, the "Terminated" message sent by the dying actor will be received by the monitoring actor asynchronously just like any other message, so the problem may still occur in between the target actor's death and the reception of its "Terminated" message.
To solve this problem, I made it so that anyone asking the supervisor for a reference to such an actor has to send a "close session" message to the supervisor to release the actor when he doesn't need it anymore (this is done transparently by a disposable "ActorSession" object). As long as there are any open sessions on an actor, the supervisor will not delete it.
I suppose that this situation is quite common and am therefore wondering if there isn't a simpler pattern to follow to address this kind of problem. Any suggestion would be appreciated.
I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception.
This is by design. You will never receive an exception upon attempting to send a message - it will simply be routed to Deadletters and logged. There's a lot of reasons for this that I won't get into here, but the bottom line is that this is intended behavior.
DeathWatch is the right tool for this job, but as you point out - you might receive a Terminated message after you already sent a message to that actor.
A simpler pattern than tracking open / closed sessions is to simply use acknowledgement / reply messages from the recipient using Ask + Wait + a hard timeout. The downside of course is that if your recipient actor has a lot of long-running operations then you might block for a long period of time inside the sender.
The other option you can go with is to redesign your recipient actor to act as a state machine and have a soft-terminated or terminating state that it uses to drain connections / references with potential senders. That way the original actor can still reply and accept messages, but let callers know that it's no longer available to do work.
I solved this problem with entity actors created through Akka's Cluster Sharding mechanism:
If the state of the entities are persistent you may stop entities that are not used to reduce memory consumption. This is done by the application specific implementation of the entity actors for example by defining receive timeout (context.setReceiveTimeout). If a message is already enqueued to the entity when it stops itself the enqueued message in the mailbox will be dropped. To support graceful passivation without losing such messages the entity actor can send ShardRegion.Passivate to its parent Shard. The specified wrapped message in Passivate will be sent back to the entity, which is then supposed to stop itself. Incoming messages will be buffered by the Shard between reception of Passivate and termination of the entity. Such buffered messages are thereafter delivered to a new incarnation of the entity.
SQS expects your application to be idempotent and I've got multiple consumers/producers where (even if SQS had a deliver-once mechanism) I will have race conditions creating duplicates and race conditions consuming because my consumers run via cron jobs.
My current plan is to use the Django 1.4 select_for_update which should block other consumers on the same row, doing something like:
reminders = EmailReminder.objects.select_for_update().filter(id=some_id)
if not reminders[0].finished:
reminder.send()
reminder.update(finished=datetime.now())
# Delete job.
Are there better ways of dealing with this?
Hook up django-celery to SQS and have it designate a periodic job using celerybeat. Then have celeryd worker(s) running on the same queue anywhere you want. Only one will pick up a job at a time and execute it. No need to introduce DB locking on any level.
As long as your worker is guaranteed to finish its current task before celerybeat fires a new one you will never have a need for a lock. Now if you think there is a chance they may overlap you can introduce states for your notifications where:
Any reminder starts in "unsent" state.
Your celerybeat sends a request to process unsent emails to the queue.
Some worker picks it up and grabs all of them.
Immediately the worker transitions all of them to "sending" state.
Proceeds to send them one at a time (or in bulk).
If sending fails for any, revert their state back to unsent.
For all that succeeded transition to sent.
This way if celerybeat fires another job while your original job is not done with the initial batch, you won't have duplicate emails sent. As an added bonus you can scale the solution and distribute the load.