My Dataflow pipeline collates event data into typed per session and per user PCollections output. I have employed GroupByKey for events keyed by session id. Sessions are grouped into parent types keyed by user id and device id using the same pattern at this next level of hierarchy. So a single session might generate many events, but in turn a single user might generate many sessions.
I would now like to summarize this data across each level of the hierarchy. I have used a StateSpec declaration to persist state at the event level. So for example, an event count property can be incremented in my event processing ParDo. (Use Case : generating an error event per session across all users for example.)
But as each ParDo is static - I cannot access the ValueState outside of the ParDo context even though my understanding is this state is maintained at the Window scope. (Maybe this is by design.) Is there a way to access this Window level state using the Beam State persistence lib in another ParDo than where it was originally declared? Like as if I could declare it at the pipeline level?
I understand that this may introduce some performance overhead as the framework must manage concurrency, but the actual processing seems negligible. (Just incrementing values.) So I would prefer to write this to a window level state field rather than percolate values up via my hierarchy.
State sharing cross ParDos is not supported, and it shouldn't even be encouraged as it brings dependencies among ParDos that breaks the simple contract: ParDo can work on PCollection independently thus unblocks massive parallelism.
Related
I am reading documentation about stateful functions in Apache Beam and I don't understand some parts of it.
I found that here if I will not assign any window to PCollection it means that global window will be used by default.
Note that even if you don’t set a windowing function, there is still a window – all elements in your PCollection are assigned to a single global window.
After that I read this article about stateful processing in Apache Beam. I found here that default parallelization for stateful function will be done per key and window.
A state cell in Beam is scoped to a key+window pair.
Am I right in case of usage of unbounded collection with ten unique keys for example I will have ten separate stateful PTransform for each key with infinite state, right?
I have an actor system that at the moment accepts commands/messages. The state of these actors is persisted Akka.Persistance. We now want to build the query system for this actor system. Basically our problem is that we want to have a way to get an aggregate/list of all the states of these particular actors. While I'm not strictly subscribing to the CQRS pattern I think that it might be a neat way to go about it.
My initial thoughts was to have an actor for querying that holds as part of its state an aggregation of the states of the other actors that are doing the "data writes". And to do this this actor will subscribe to the actors its interested to and these actors would just send the query actor their states when they undergo some sort of state change. Is this the way to go about this? is there a better way to do this?
My recommendation for implementing this type of pattern is to use a combination of pub-sub and push-and-pull messaging for your actors here.
For each "aggregate," this actor should be able to subscribe to events from the individual child actors you want to query. Whenever a child's state changes, a message is pushed into all subscribed aggregates and each aggregate's state is updated automatically.
When a new aggegrate comes online and needs to retrieve state it missed (from before it existed) it should be able to pull the current state from each child and use that to build its current state, using incremental updates from children going forward to keep its aggregated view of the children's state consistent.
This is the pattern I use for this sort of work and it works well locally out of the box. Over the network, you may have to ensure deliverability guarantees and that's generally easy to do. You can read a bit more on how to do that there: https://petabridge.com/blog/akkadotnet-at-least-once-message-delivery/
Some of Akka.Persistence backends (i.e. those working with SQL) also implement something known as Akka.Persistence.Query. It allows you to subscribe to a stream of events that are produced, and use this as a source for Akka.Streams semantics.
If you're using SQL-journals you'll need Akka.Persistence.Query.Sql and Akka.Streams packages. From there you can create a live (that means continuously updated) source of events for a particular actor and use it for any operations you like i.e print them:
using (var system = ActorSystem.Create("system"))
using (var materializer = system.Materializer())
{
var queries = Sys.ReadJournalFor<SqlReadJournal>(SqlReadJournal.Identifier)
queries.EventsByPersistenceId("<persistence-id>", 0, long.MaxValue)
.Select(envelope => envelope.Event)
.RunForEach(e => Console.WriteLine(e), materializer);
}
I'm trying to implement this pattern on a "smart building" system design (using STL library). Various "sensors" placed in rooms, floors etc, dispatch signals that are handled by "controllers" (also placed in different rooms, floors etc.). The problem I'm facing is that the controller's subscription to an event isn't just event based, it is also location based.
For example, controller A can subscribe to a fire signal from room #1 in floor #4 and to a motion signal in floor #5. A floor-based subscription means that controller A will get an motion event about every room in the floor he's subscribed to (assuming the appropriate sensor is placed there). There's also a building-wide subscription for that matter.
The topology of the system is read from a configuration file at start up, so I don't want to map the whole building, just the relevant places that contain sensors and controllers.
What I've managed to think of :
Option 1: MonitoredArea class that contains the name of the area (Building1, Floor 2, Room 3) and a vector where the vector's index is an enumerated event type each member of the vector contains a list of controllers that are subscribed to this event. The class will also contain a pointer to a parent MonitoredArea, in the case it is a room in a floor, or a floor in a building.
A Sensor class will dispatch an Event to a center hub along with the sensor's name. The hub will run it through his sensor-name-to-location map, acquire the matching MonitoredArea and will alert all the controllers in the vector.
Cons:
Coupling of the location to the controller
Events are enumerated and are hard coded in the MonitoredArea class, adding future events is difficult.
Option 2:
Keeping all the subscriptions in the Controller class.
Cons:
Very inefficient. Every event will make the control center to iterate through all the controller and find out which are subscribed to this particular event.
Option 3:
Event based functionality. Event class (ie. FireEvent) will contain all the locations it can happen in (according to the sensor's setup) and for every location, a list of the controllers that are subscribed to it.
Cons:
A map of maps
Strong data duplication
No way to alert floor-based subscriptions about events in the various rooms.
As you can see, I'm not happy with any of the mentioned solutions. I'm sure I've reached the over-thinking stage and would be happy for a feedback or alternative suggestions as to how I approach this. Thanks.
There is design pattern (sort of speak) used a lot in game development called "Message Bus". And it is sometimes used to replace event based operations.
"A message bus is a connection between one or more senders and/or receivers. Think of it like a connection between computers in a bus topology: Every node can send a message by passing it to the bus, and all connected nodes will receive that message. If the node is processed and if a reply is sent is completely up to each receiver itself.
Having modules connected to a message bus gives us some advantages:
Every module is isolated, it does not need to know of any others.
Every module can react to any message that’s being sent to the bus; that means you get extra flexibility for free, without increasing dependencies at all.
It’s much easier to follow the YAGNI workflow: For example you’re going to add weapons. At first you implement the physics, then you add visuals in the renderer, and then playing sounds. All of those features can be implemented independently at any time, without interrupting each other.
You save yourself from thinking a lot about how to connect certain modules to each other. Sometimes it takes a huge amount of time, including drawing diagrams/dependency graphs."
Sources:
http://gameprogrammingpatterns.com/event-queue.html
http://www.optank.org/2013/04/02/game-development-design-3-message-bus/
Let's say I have a command to edit a single entry of an article, called ArticleEditCommand.
User 1 issues an ArticleEditCommand based on V1 of the article.
User 2 issues an ArticleEditCommand based on V1 of the same
article.
If I can ensure that my nodes process the older ArticleEditCommand commands first, I can be sure that the command from User 2 will fail because User 1's command will have changed the version of the article to V2.
However, if I have two nodes process ArticleEditCommand messages concurrently, even though the commands will be taken of the queue in the correct order, I cannot guarantee that the nodes will actually process the first command before the second command, due to a spike in CPU or something similar. I could use a sql transaction to update an article where version = expectedVersion and make note of the number of records changed, but my rules are more complex, and can't live solely in SQL. I would like my entire logic of the command processing guaranteed to be concurrent between ArticleEditCommand messages that alter that same article.
I don't want to lock the queue while I process the command, because the point of having multiple command handlers is to handle commands concurrently for scalability. With that said, I don't mind these commands being processed consecutively, but only for a single instance/id of an article. I don't expect a high volume of ArticleEditCommand messages to be sent for a single article.
With the said, here is the question.
Is there a way to handle commands consecutively across multiple nodes for a single unique object (database record), but handle all other commands (distinct database records) concurrently?
Or, is this a problem I created myself because of a lack of understanding of CQRS and concurrency?
Is this a problem that message brokers typically have solved? Such as Windows Service Bus, MSMQ/NServiceBus, etc?
EDIT: I think I know how to handle this now. When User 2 issues the ArticleEditCommand, an exception should be throw to the user letting them know that there is a current pending operation on that article that must be completed before then can queue the ArticleEditCommand. That way, there is never two ArticleEditCommand messages in the queue that effect the same article.
First let me say, if you don't expect a high volume of ArticleEditCommand messages being sent, this sounds like premature optimization.
In other solutions, this problem is usually not solved by message brokers, but by optimistic locking enforced by the persistence implementation. I don't understand why a simple version field for optimistic locking that can be trivially handled by SQL contradicts complicated business logic/updates, maybe you could elaborate more?
It's actually quite simple and I did that. Basically, it looks like this ( pseudocode)
//message handler
ModelTools.TryUpdateEntity(
()=>{
var entity= _repo.Get(myId);
entity.Do(whateverCommand);
_repo.Save(entity);
}
10); //retry 10 times until giving up
//repository
long? _version;
public MyObject Get(Guid id)
{
//query data and version
_version=data.version;
return data.ToMyObject();
}
public void Save(MyObject data)
{
//update row in db where version=_version.Value
if (rowsUpdated==0)
{
//things have changed since we've retrieved the object
throw new NewerVersionExistsException();
}
}
ModelTools.TryUpdateEntity and NewerVersionExistsException are part of my CavemanTools generic purpose library (available on Nuget).
The idea is to try doing things normally, then if the object version (rowversion/timestamp in sql) has changed we'll retry the whole operation again after waiting a couple of miliseconds. And that's exactly what the TryUpdateEntity() method does. And you can tweak how much to wait between tries or how many times it should retry the operation.
If you need to notify the user, then forget about retrying, just catch the exception directly and then tell the user to refresh or something.
Partition based solution
Achieve node stickiness by routing the incoming command based on the object's ID (eg. articleId modulo your-number-of-nodes) to make sure the commands of User1 and User2 ends up on the same node, then process the commands consecutively. You can choose to process all commands one by one or if you want to parallelize the execution, partition the commands on something like ID, odd/even, by country or similar.
Grid based solution
Use an in-memory grid (eg. Hazelcast or Coherence) and use a distributed Executor Service (http://docs.hazelcast.org/docs/2.0/manual/html/ch09.html#DistributedExecution) or similar to coordinate the command processing across the cluster.
Regardless - before adding this kind of complexity, you should of course ask yourself if it's really a problem if User2's command would be accepted and User1 got a concurrency error back. As long as User1's changes are not lost and can be re-applied after a refresh of the article it might be perfectly fine.
How to handle concurrency by eventual consistency? Or I could ask how to ensure data integrity by eventual consistency?
By CQRS and event sourcing, eventual consistency means, that you put your domain events into a queue, and you set event handlers which are projections. Those projections update the read cache in an async way. Now if you validate using that read cache, you cannot be sure that the information you base your validation on, is still valid. There can be unprocessed (or unprojected?) domain events in the queue when you send your command, which can change the outcome of the validation. So this is just another type of concurrency... What do you think, how to handle these rare concurrency issues? Domain events are already saved in the storage, so you cannot do anything about them, you cannot just remove them from the event storage (because it supposed to be write only once), and tell the user in an email, that sorry, we made up our mind and cancelled your request. Or can you?
update:
A possible solution to handle concurrency by an event storage:
by write model
if
last-known-aggregate-version < stored-aggregate-version
then
throw error
else
execute command on aggregate
raise domain-event
store domain-event
++stored-aggregate-version (by aggregate-id)
by read model
process query
if
result contains aggregate-id
then
attach read-cached-aggregate-version
by projection
process domain-event
read-cached-aggregate-version = domain-event-related-aggregate-version (by aggregate-id)
As long as state changes you cannot assume anything will ever be 100% consistent. Technically you can ensure that various bits are 100% consistent with what you know.
Your queued domain event scenario is no different from a queue of work on a user's desk that still has to be input into the system.
Any other user performing an action dependent on the system state has no way to know that another user still needs to perform some action that may interfere with their operation.
I guess a lot is based on assuming the data is consistent and developing alternate flows and processes that can deal with these scenarios as they arise.