I'm working on a data import job using the Go language, I want to write each step as a closure, and use channels for communication, that is, each step is concurrent. The problem can be defined by the following structure.
Get Widgets from data source
Add translations from source 1 to Widgets.
Add translations from source 2 to Widgets.
Add pricing from source 1 to Widgets.
Add WidgetRevisions to Widgets.
Add translations from source 1 to WidgetRevisions
Add translations from source 2 to WidgetRevisions
For the purposes of this question, I'm only dealing with the first three steps which must be taken on a new Widget. I assume on that basis that step four could be implemented as a pipeline step, which in itself is implemented in terms of a sub-three-step pipeline to control the *WidgetRevision*s
To that end I've been writing a small bit of code to give me the following API:
// A Pipeline is just a list of closures, and a smart
// function to set them all off, keeping channels of
// communication between them.
p, e, d := NewPipeline()
// Add the three steps of the process
p.Add(whizWidgets)
p.Add(popWidgets)
p.Add(bangWidgets)
// Start putting things on the channel, kick off
// the pipeline, and drain the output channel
// (probably to disk, or a database somewhere)
go emit(e)
p.Execute()
drain(d)
I've implemented it already ( code at Gist or at the Go Playground) but it's deadlocking with a 100% success failure rate
The deadlock comes when calling p.Execute(), because presumably one of the channels is ending up with nothing to do, nothing being sent on any of them, and no work to do...
Adding a couple of lines of debug output to emit() and drain(), I see the following output, I believe the pipelining between the closure calls is correct, and I'm seeing some Widgets being omitted.
Emitting A Widget
Input Will Be Emitted On 0x420fdc80
Emitting A Widget
Emitting A Widget
Emitting A Widget
Output Will Drain From 0x420fdcd0
Pipeline reading from 0x420fdc80 writing to 0x420fdd20
Pipeline reading from 0x420fdd20 writing to 0x420fddc0
Pipeline reading from 0x420fddc0 writing to 0x42157000
Here's a few things I know about this approach:
I believe it's not uncommon for this design to "starve" one coroutine or another, I believe that's why this is deadlocking
I would prefer if the pipeline had things fed into it in the first place (API would implement Pipeline.Process(*Widget)
If I could make that work, the drain could be a "step" which just didn't pass anything on to the next function, that might be a cleaner API
I know I haven't implemented any kind of rung buffers, so it's entirely possible that I'll just overload the available memory of the machine
I don't really believe this is good Go style... but it seems to make use of a lot of Go features, but that isn't really a benefit
Because of the WidgetRevisions also needing a pipeline, I'd like to make my Pipeline more generic, maybe an interface{} type is the solution, I don't know Go well enough to determine if that'd be sensible or not yet.
I've been advised to consider implementing a mutex to guard against race conditions, but I believe I'm save as the closures will each operate on one particular unit of the Widget struct, however I'd be happy to be educated on that topic.
In Summary: How can I fix this code, should I fix this code, and if you were a more experienced go programmer than I, how would you solve this "sequential units of work" problem?
I just don't think I would've built the abstractions that far away from the channels. Pipe explicitly.
You can pretty easily make a single function for all of the actual pipe manipulation, looking something like this:
type StageMangler func(*Widget)
func stage(f StageMangler, chi <-chan *Widget, cho chan<- *Widget) {
for widget := range chi {
f(widget)
cho <- widget
}
close(cho)
}
Then you can pass in func(w *Widget) { w.Whiz = true} or similar to the stage builder.
Your add at that point could have a collection of these and their worker counts so a particular stage could have n workers a lot more easily.
I'm just not sure this is easier than piecing channels together directly unless you're building these pipelines at runtime.
Related
Using akka-typed I'm trying to create an event-sourced application in which a command on an actor can cause effect on another actor. Specifically I have the following situation:
RootActor
BranchActor (it's the representation of a child of the Root)
When RootActor is issued a CreateBranch command, validation happens, and if everything is o.k. the results must be:
RootActor will update the list of children of that actor
BranchActor will be initialized with some contents (previously given in the command)
RootActor replies to the issuer of the command with OperationDone
Right now the only thing I could come up with is: RootActor processes the Event and as a side effect issues a command to the BranchActor, which in turn saves an initialization eventt, replies to the RootActor, which finally replies to the original issuer.
This looks way too complicated, though, because:
I need to use a pipe to self mechanism, which implies that
I need to manage internal commands as well that allow me to reply to the original issuer
I need to manage the case where that operation might fail, and if this fails, it means that the creation of a branch is not atomic, whereas saving two events is atomic, in the sense that either both are saved or neither is.
I need to issue another command to another actor, but I shouldn't need to do that, because the primary command should take care of everything
The new command should be validated, though it is not necessary because it comes from the system and not an "external" user in this case.
My question then is: can't I just save from the RootActor two events, one for self, and one for a target BranchActor?
Also, as a bonus question: is this even a good practice for event-sourcing?
My question then is: can't I just save from the RootActor two events, one for self, and one for a target BranchActor?
No. Not to sound trite, but the only thing you can do to an actor is to send a message to it. If you must do what you are doing you are doing, you are on the right path. (e.g. pipeTo etc.)
is this even a good practice for event-sourcing?
It's not a good practice. Whether it's suboptimal or a flat out anti-pattern is still debatable. (I feel like I say say this confidently because of this Lightbend Discussion thread where it was debated with one side arguing "tricky but I have no regrets" and the other side arguing "explicit anti-pattern".)
To quote someone from an internal Slack (I don't want attribute him without his permission, but I saved it because it seemed to so elegantly sum up this kind of scenario.)
If an event sourced actor needs to contact another actor to make the decision if it can persist an event, then we are not modeling a consistency boundary anymore. It should only rely on the state that [it has] in scope (own state and incoming command). … all the gymnastics (persist the fact that its awaiting confirmation, stash, pipe to self) to make it work properly is an indication that we are not respecting the consistency boundary.
If you can't fix your aggregates such that one actor is responsible for the entire consistency boundary the better practice is to enrich the command beforehand: essentially building a Saga pattern.
Got really stuck, need some advise or real examples.
1) I have boost::thread vector producer thread (data arrives fast ~ 100 samples per second)
2) I want QMathGL to paint data as it arrives
3) I don't want my Qt gui freeze
I tried to move QMathGL::update() to separate thread - Qt argues that QPixmap not allowed in separate thread.
What should i try, Without modifying QMathGL?
Only thing comes in mind to repaint on timer (fps?), but i don't like this solution, please tell me if i am wrong.
I would strongly advise to go with a timer. Repaint operations are costly and I would assume that no user could realistically process more then 10 printed vectors a second. So I can't see a real benefit for the end user, apart from maybe that the display is updated more "smoothly" and entry for entry. But you could achieve these effects far easier with animations ;)
When repainting with every data change, you get the annoying behaviour you describe. Working around that is (imho) not worth the trouble.
I´ve also come along a similar problem sometimes.
The usual resolution i used is to buffer the data and repainting on a timer. This goes along the line of this (Pseudo Code) :
void Widget::OnNewData(void *dataSample)
{
this->threadSafebuffer->appendData(dataSample);
}
void Widget::OnTimeout()
{
DataBuffer renderBatch = this->threadSafebuffer->interlockedExchange();
/* Do UI updates according to renderBatch */
}
This assumes that OnNewData is called on a background thread. The OnTimeout is called from a QTimer on the UI-EventLoop. To prevent contention it justs does an interlocked exchange of the current buffer pointer with a second buffer. So no heavy synchronization (e.g. Mutext/Semaphore) is needed.
This will only work if the amount of work to do for rendering a renderBatch is less than the timeout.
I'm reading values from a sensor via serial (accelerometer values), in a loop similar to this:
while( 1 ) {
vector values = getAccelerometerValues();
// Calculate velocity
// Calculate total displacement
if ( displacement == 0 )
print("Back at origin");
}
I know the time that it takes per sample, which is taken care of in the getAccelerometerValues(), so I have a time-period to calculate velocity, displacement etc. I sample at approximately 120 samples / second.
This works, but there are bugs (non-precise accelerometer values, floating-point errors, etc), and calibrating and compensating to get reasonably accurate displacement values is proving difficult.
I'm having great amounts of trouble finding a process to debug the loop. If I use a debugger (my code happens to be written in C++, and I'm slowly learning to use gdb rather than print statements), I have issues stepping through and pushing my sensor around to get an accelerometer reading at the point in time that the debugger executes the line. That is, it's very difficult to get the timing of "continue to next line" and "pushing sensor so it's accelerating" right.
I can use lots of print statements, which tend to fly past on-screen, but due to the number of samples, this gets tedious and difficult to derive where the problems are, particularly if there's more than one print statement per loop tick.
I can decrease the number of samples, which improves the readability of the programs output, but drastically decreases the reliability of the acceleration values I poll from the sensor; if I poll at 1Hz, the chances of polling the accelerometer value while it's undergoing acceleration drop considerably.
Overall, I'm having issues stepping through the code and using realistic data; I can step through it with non-useful data, or I can not really step through it with better data.
I'm assuming print statements aren't the best debugging method for this scenario. What's a better option? Are there any kinds of resources that I would find useful (am I missing something with gdb, or are there other tools that I could use)? I'm struggling to develop a methodology to debug this.
A sensible approach would be to use some interface for getAccelerometerValues() that you can substitute at either runtime or build-time, such as passing in a base-class pointer with a virtual method to override in the concrete derived class.
I can describe the mechanism in more detail if you need, but the ideal is to be able to run the same loop against:
real live data
real live data (and save it to file)
canned real data saved from a previous run
fake data you cooked up as a test case
Note in particular that the 'replay' version(s) should be easy to debug, if each call just returns the next data from the file.
Create if blocks for the exact conditions you want to debug. For example, if you only care about when the accelerometer reads that you are moving left:
if(movingLeft(values) {
print("left");
}
The usual solution to this problem is recording. Your capture sample sequences from your sensor in a real-time manner, and store them to files. Then you train your system, debug your code, etc. using the recorded data. Finally, you connect the working code to the real data stream which flows immediately from the sensor.
I would debug your code with fake (ie: random) values, before everything else.
It the computations work as expected, then I would use the values read from the port.
Also, isnt' there a way to read those values in a callback/push fashion, that is, get your function called only when there's new, reliable data?
edit: I don't know what libraries you are using, but in the .NET framework you can use a SerialPort class with the Event DataReceived. That way you are sure to use the most actual and reliable data.
I am trying to make some design decisions for an algorithm I am working on. I think that I want to use signals and slots to implement an observer pattern, but I am not sure of a few things.
Here is the algorithm I am working towards:
1.) Load tiles of an image from a large file
1a.) Copy the entire file to a new location
2.) Process the tiles as they are loaded
3.) If the copy has been created, copy the resulting data into the new file
so I envision having a class with functions like loadAllTiles() which would emit signals to tell a processTile() that another tile was ready to be processed, while moving on to load the next tile.
processTile() would perform some calculations, and when complete, signal to writeResults() that a new set of results data was ready to be written. writeResults() would verify that the copying was complete, and start writing the output data.
Does this sound reasonable? is there a way to make loadAllTiles() load in a tile, pass that data somehow to processTile() and then keep going and load the next tile? I was thinking about maybe setting up a list of some sort to store the tiles ready to be processed, and another list for result tiles ready to be written to disk. I guess the disadvantage there is I have to somehow keep those lists in tact, so that multiple threads arent trying to add/remove items from the list.
Thanks for any insight.
It's not completely clear in your question, but it seems that you want to split up the work into several threads, so that the processing of tiles can begin before you finishing loading the entire set.
Consider a multithreaded processing pipeline architecture. Assign one thread per task (loading, copying, processing), and pass around tiles between tasks via Producer-Consumer queues (aka BlockingQueue). To be more precise, pass around pointers (or shared pointers) to tiles to avoid needless copying.
There doesn't seem to be a ready-made thread-safe BlockingQueue class in Qt, but you can roll-up your own using QQueue, QWaitCondition, and QMutex. Here are some sources of inspiration:
Just Software Solutions' blog article.
Java's BlockingQueue
ZThreads's BlockingQueue
While there isn't a ready-made BlockingQueue within Qt, it seems that using signals & slots with the Qt::QueuedConnection option may serve the same purpose. This Qt blog article makes such use of signals and slots.
You may want to combine this pipeline approach with a memory pool or free list of tiles, so that already allocated tiles are recycled in your pipeline.
Here's a conceptual sketch of the pipeline:
TilePool -> TileLoader -> PCQ -> TileProcessor -> PCQ -> TileSaver -\
^ |
\----------------------------------------------------------------/
where PCQ represents a Producer-Consumer queue.
To exploit even more parallelism, you can try thread pools at each stage.
You can also consider checking out Intel's Threading Building Blocks. I haven't tried it myself. Be aware of the GPL licence for the open source version.
Keeping the lists from corruption should be possible with any kind of parallelization lock mechanisms, be it simple locks, semaphores etc etc.
Otherwise, the approach sounds reasonable, even though I would say that the files had to be large in order for this to make sense. As long as they easily fit into memory, I don't see the point of loading them piecewise. Also: How do you plan on extracting tiles without reading the entire image repeatedly?
I'v always wondered how this works. Does having more slow down the game? for example how would one represent checking if a car has flipped. It could be seen as:
if (player.car.angle.y == 180)
{
do something
}
The parts that puzzle me is, when would the game check for it? The way I see it, every thing that can happen in the game seems to evolve an if. I'm just wondering how these are handled.
Also, since the game runs through a loop, what if the car is flipped for more than 1 frame, would a Boolean really be used to check if the event has been fired
Thanks
In most general terms, any object in an engine has a state - if it changes state (e.g. not flipped to flipped), that is a transition.
From a transition you can fire an event or not, but as the transition does only occur when changing state the event won't be fired more then once.
As for the conditions that trigger the transitions, they have to be coded somewhere of course. Sometimes they are more explicitly coded, but mostly they are parameterized so that scripts or some sort of configuration can change them easily.
How it is implemented in the end differs broadly, it depends on the libraries that are used as well the engine design itself.