It seems an obvious combination to use django's signalling architecture together with channels: a model is changed, a signal is fired, and a consumer is notified (via the signal handler and channel layer) to update a client via, for instance, a WebSocket.
However, very often a model is updated as part of a transaction, and django's pre_save and post_save signals are sent in the middle of the transaction, meaning the change you notify the client about may never happen at all. Even more annoyingly, when you send the information across the channel layer you're looking at the database from another thread and, since the transaction is not complete, you can't get the new data.
django.db.transaction.on_commit provides a way to schedule (part of) the signal handler to run after the transaction is done, providing a workaround, but this seems to be such an obvious problem with absolutely no documentation or stackoverflow chatter that I'm a bit perturbed (one can of course find discussions about each pair of the three mentioned topics!) So is this architecture: signal -> handler -> channel layer -> consumer [ -> client ] actually a bad one? Is there some alternative that doesn't involve wrapping each handler in a on_commit hook, or is there some common idiom for doing that an DRY way?
Related
I have a dispatcher thread and a listener thread. When I dispatch a command, I want to wait for response before I send follow up command. Moreover I need to examine the respond before I can proceed with 2nd command, the least of which is to confirm the response is received and everything is okay. My pseudo code is below:
void MainWindow::downloadData()
{
dispatcher->getInfo(); // sends command
// QString response = receiver->response() // idealy I would like to check response but since its async, i can't really do that!
dipatcher->askData(); // the 2nd command and so forth
}
Is there any elegant way to solve this issue? The only way I can think of is if I use the same thread and all calls are blocking but that's not necessarily a good solution.
In Qt, I could use signals and slots to connect them in cascading manner so when the first signal is triggered it initiates the whole sequence of operation (each slot emitting a new signal) but seems rather dirty as well.
One of the most robust ways to handle asynchronous events and process a chains/graphs of actions upon these events are FSMs. Qt provides a basis for implementing FSMs with its Qt-State machine framework. I'd suggest to go this way. Unfortunately all the examples provided by Qt for FSM are dealing with GUIs and animations.
The advantage of FSM approach is, FSMs can be represented both as graphs and as tables. The first option is great for understanding, the second for validation, that there are no endless loops and "dead" ends.
I've built on basis of Qt-FSM framework own framework for defining FSMs in a domain specific language. I use it for controlling a complex machine having couple of sensors actors all working asynchronously. Using DSL helps me to implement in higher abstraction - in the abstraction level of FSM-graphs.
I have three actors to handle a CQRS scenario. The ProductWorkerActor handles the command part, ProductQueryWorkerActor handles the query part and ProductStateActor handles the state part.
The way I'm handling the query is by using:
ProductQueryWorkerActor.Ask<ProductState>("give-me-product-state-for-product-1000")
The code from ProductQueryWorkerActor:
if (message == "give-me-product-state-for-product-1000")
{
var actor = Context.ActorSelection("akka://catalogSystem/user/productState/1000");
var psDTO = actor.Ask<ProductStateDTO>(message).Result;
Sender.Tell(ps);
}
Please ignore the path being used to access the product state. It is hardcoded and intentional to make the code read simpler.
Should I be using Ask as I have used in this case to retrieve the state of a product? Is Ask called Futures?
Should I be exposing the state as DTO to the outside work instead of the actor itself?
To change any state of the product, should I handle the message processing in ProductWorkerActor or in ProductStateActor itself? In the second case, the ProductWorkerActor sends a message to ProductStateWorker, the ProductStateWorker processes the message, change the state and send another message to ProductWorkerActor that it passed validation and changed the state.
In case when you're using Event Sourcing with your actors, I advice you to use Akka.Persistence. It handles read/write actors separation and will take a lot of burden from you shoulders.
If not, in my opinion basic problem with your design is that, while you have separate actors for reading/writing to state, state itself is handled in only one actor. Why? One of the points of CQRS is to have a separate models optimized for serving their role (either read or write).
In example: you can have one handler actor (eg. ProductActor) changing it's state based on incoming commands, and bunch of different readonly actors (eg. ProductHistoryActor, ProductListActor), each with it's own state optimized for their role. Readonly actors may subscribe to event stream to listen for incoming messages about handler actor's state changes and updating their own states accordingly, while handler actor after handling a command publishes message about state change using actor system's event stream.
Ad. 1: In my opinion using Ask to communicate between actors is an anti-pattern. In your example, you're using query actor to pass message through to state actor, then blocking current actor until response arrives (which is very bad for performance) just to send message back to sender. Instead of using:
var psDTO = actor.Ask<ProductStateDTO>(message).Result;
Sender.Tell(ps);
you could simply write:
actor.Forward(message);
and let actor send response directly to sender (you query actor doesn't need to participate with sending the response).
Ad. 2: It depends on your case, but remember - you should never pass mutable objects as messages, especially when you use them after sending.
Ad. 3: I think that in your example distinction between ProductWorkerActor and ProductStateWorker is artificial. From what you're showing, they should be a single entity IMO.
We're using Qt 4.8.2, and we have a model/view design (subclassed QAbstractItemModel and QTreeview, specifically). The model/treeview follows the typical philosophy where the view drives the model - we don't populate the model until the user expands the corresponding treeview nodes.
Once a node has been expanded & data is visible, it is subject to display updates that occur in worker (non-UI) threads. Right now, when a worker thread produces a change that may affect the treeview, it emits a "change" signal, which maps to a slot in our model.
The problem is that these change signals can be emitted with great frequency (say, 1500 events over a second), sometimes, but they may apply to what the treeview currently displays (and can therefore be ignored). When this happens, the UI thread becomes unresponsive as (I presume) the signals all queue up and the UI thread has to deal with them before it can resume responding to user interaction.
The time it takes to respond to a change signal is very small, but it appears that the UI thread only "eats" the signals after a small delay - presumably to avoid excessive updating resulting in screen flicker or other annoyances.
The result is that the UI remains frozen for 5 or 6 seconds, with very low CPU activity during this time (presumably because the signals are coming in fast enough the handler is still waiting for break in the action); once all the signals are queued up, the thread finally consumes the work in the queue and resolves it in a few milliseconds.
I have several thoughts on this:
Is there some setting such that I can increase the aggressiveness by which the UI thread handles incoming signals?
Is it feasible at all to manage the updates of the model in a separate thread? My instinct is to say no - it would seem the Qt machinery is too dependent on exclusive ownership of the model, and putting the appropriate lock protection around its access would be complicated and violate the whole point of the slot/signal paradigm.
I can think up more complex schemes to deal with this signals in a secondary thread; for example, the UI could maintain a separate multithread-visible (non-model) data structure that could be queried to determine whether a change signal needed to be sent. Similarly, I could maintain a separate queue that the worker threads use, where I could batch up change events into a single signal (which I could deliver no more than twice a second, for example). But these methods strike me as a bit byzantine for a problem that I assume must not be totally uncommon in the domain of Qt UI programming.
We had a similar application with large updates to underlying data. The question comes down to:
1500 updates per second will result in how many changes in the GUI?
If the answer is there will be less than 6 changes, then the model should only emit 6 data changes per second. If this is the case, when underlying data changed, check if this change will change the GUI or not, only emit data changed signal from model when necessary.
If the answer is there will be more than 6 gui changes per second, the answer we have is no human being can see more then 3 changes per second. The underlying data change should not update GUI at all. Use a 250 milli second timer, in the timer event, check the cells needs to be updated, and update them.
I have a situation where I have a single Emitter object and a set of Receivers. The receivers are of the same class, and actually represent a set of devices of the same type. I'm using the Qt framework.
The Emitter itself first gets a signal asking for information from one of the devices.
In the corresponding slot, the Emitter has to check to see which of the Receivers are 'ready', and then send its own signal to request data to one of the devices (whichever is ready first).
The Emitter receives signals very quickly, on the order of milliseconds. There are three ways I can think of safely requesting data from only one of the devices (the devices live in their own threads, so I need a thread-safe mechanism). The number of devices isn't static, and can change. The total number of devices is quite small (definitely under 5-6).
1) Connect to all the devices when they are added or removed. Emit the one request and have the devices objects themselves filter out whether the request is for them using some specific device tag. This method is nice because the request slot where the check occurs will execute in a dedicated thread's context, but wasteful as the number of devices go up.
2) Connect and disconnect from the object within the Emitter on the fly when it's necessary to send a request.
3) Use QMetaObject::invokeMethod() when its necessary to send a request.
Performance is important. Does anyone know which method is the 'best', or if there's a better one altogether?
Regards
Pris
Note: To clarify: Emitter gets a signal from the application, to get info by querying the device. Crazy ASCII art go:
(app)<---->(emitter)<------>(receivers)<--|-->physical devices
Based on the information you have provided I would still recommend a Reactor implementation. If you don't use ACE then you can implement your own. The basic architecture is as follows:
use select to wake up when signal or data is received from the App.
If there is a socket ready on the sending list then you just pick one and send it data
When data is sent the Receiver removes itself from the set of sockets/handlers that are available
When data is processed the Reciever re-registers itself to the list of available recipients.
The reason I suggested ACE is because it has one of the simplest to use implementations of the Reactor pattern.
I'm amusing here this is multi thread environment.
If you are restricted to Qt signal / slot system between then the answer for your specific questions:
1) is definitely not the way to go. On an emit from the Emitter a total number of events equal to the number of Receivers will be queued for the thread(s) event loops of the devices, then the same number of slot calls will occur once the thread(s) reach those event. Even if most of the lost just if(id!=m_id) return; on their first line, its a significant amount of things going on in the core of Qt. Place a breakpoint in one of your slots that is evoked by a Qt::QueuedConnection signal and validate this looking at the actual stack trace. Its usually at least 4 call deep from the xyEventLoop::processEvents(...), so "just returning" is definitely not "free" in terms of time.
2) Not sure how Qt's inner implementation actually is, but from what I know connecting and disconnecting most likely include inserting and removing the sender and receiver into some lists, which are most likely accessed with QMutex locking. - might also be "expensive" time-wise, and rapidly connecting and disconnecting is definitely not a best practice.
3) Probably the least "expensive time-wise" solution you can find that is still using Qt's singnal-slot system.
optionally) Take a look at QSignalMapper. It is designed exactly for what you planned to do in option 1).
There are more optimal solutions to communicate between your Emitter and Receivers, but as a best practice I'd first choose the option that is most easy to use and fast to implement, yet has a chance of being fast enough run-time (that is option 3). ). Then, when its done, see if it meets your performance requirements. If it does not, and only then, consider using shared memory with mutexes in a data provider - data consumer architecture (Emitter thread rapidly post request data in a circular list, while the Receiver thread(s) reads them whenever have time, then post results back a similar way, while the Emitter thread constantly polls for done results.)
I'm using Qt framework which has by default non-blocking I/O to develop an application navigating through several web pages (online stores) and carrying out different actions on these pages. I'm "mapping" specific web page to a state machine which I use to navigate through this page.
This state machine has these transitions;
Connect, LogIn, Query, LogOut, Disconnect
and these states;
Start, Connecting, Connected, LoggingIn, LoggedIn, Querying, QueryDone, LoggingOut, LoggedOut, Disconnecting, Disconnected
Transitions from *ing to *ed states (Connecting->Connected), are due to LoadFinished asynchronous network events received from network object when currently requested url is loaded. Transitions from *ed to *ing states (Connected->LoggingIn) are due to events send by me.
I want to be able to send several events (commands) to this machine (like Connect, LogIn, Query("productA"), Query("productB"), LogOut, LogIn, Query("productC"), LogOut, Disconnect) at once and have it process them. I don't want to block waiting for the machine to finish processing all events I sent to it. The problem is they have to be interleaved with the above mentioned network events informing machine about the url being downloaded. Without interleaving machine can't advance its state (and process my events) because advancing from *ing to *ed occurs only after receiving network type of event.
How can I achieve my design goal?
EDIT
The state machine I'm using has its own event loop and events are not queued in it so could be missed by machine if they come when the machine is busy.
Network I/O events are not posted directly to neither the state machine nor the event queue I'm using. They are posted to my code (handler) and I have to handle them. I can forward them as I wish but please have in mind remark no. 1.
Take a look at my answer to this question where I described my current design in details. The question is if and how can I improve this design by making it
More robust
Simpler
Sounds like you want the state machine to have an event queue. Queue up the events, start processing the first one, and when that completes pull the next event off the queue and start on that. So instead of the state machine being driven by the client code directly, it's driven by the queue.
This means that any logic which involves using the result of one transition in the next one has to be in the machine. For example, if the "login complete" page tells you where to go next. If that's not possible, then the event could perhaps include a callback which the machine can call, to return whatever it needs to know.
Asking this question I already had a working design which I didn't want to write about not to skew answers in any direction :) I'm going to describe in this pseudo answer what the design I have is.
In addition to the state machine I have a queue of events. Instead of posting events directly to the machine I'm placing them in the queue. There is however problem with network events which are asynchronous and come in any moment. If the queue is not empty and a network event comes I can't place it in the queue because the machine will be stuck waiting for it before processing events already in the queue. And the machine will wait forever because this network event is waiting behind all events placed in the queue earlier.
To overcome this problem I have two types of messages; normal and priority ones. Normal ones are those send by me and priority ones are all network ones. When I get network event I don't place it in the queue but instead I send it directly to the machine. This way it can finish its current task and progress to the next state before pulling the next event from the queue of events.
It works designed this way only because there is exactly 1:1 interleave of my events and network events. Because of this when the machine is waiting for a network event it's not busy doing anything (so it's ready to accept it and does not miss it) and vice versa - when the machine waits for my task it's only waiting for my task and not another network one.
I asked this question in hope for some more simple design than what I have now.
Strictly speaking, you can't. Because you only have state "Connecting", you don't know whether you need top login afterwards. You'd have to introduce a state "ConnectingWithIntentToLogin" to represent the result of a "Connect, then Login" event from the Start state.
Naturally there will be a lot of overlap between the "Connecting" and the "ConnectingWithIntentToLogin" states. This is most easily achieved by a state machine architecture that supports state hierarchies.
--- edit ---
Reading your later reactions, it's now clear what your actual problem is.
You do need extra state, obviously, whether that's ingrained in the FSM or outside it in a separate queue. Let's follow the model you prefer, with extra events in a queue. The rick here is that you're wondering how to "interleave" those queued events vis-a-vis the realtime events. You don't - events from the queue are actively extracted when entering specific states. In your case, those would be the "*ed" states like "Connected". Only when the queue is empty would you stay in the "Connected" state.
If you don't want to block, that means you don't care about the network replies. If on the other hand the replies interest you, you have to block waiting for them. Trying to design your FSM otherwise will quickly lead to your automaton's size reaching infinity.
How about moving the state machine to a different thread, i. e. QThread. I would implent a input queue in the state machine so I could send queries non blocking and a output queue to read the results of the queries. You could even call back a slotted function in your main thread via connect(...) if a result of a query arrives, Qt is thread safe in this regard.
This way your state machine could block as long as it needs without blocking your main program.
Sounds like you just want to do a list of blocking I/O in the background.
So have a thread execute:
while( !commands.empty() )
{
command = command.pop_back();
switch( command )
{
Connect:
DoBlockingConnect();
break;
...
}
}
NotifySenderDone();