I have a Microservice application in which I have a set of actors with each actor acting as a digital twin for say a Power producing unit. This actor gets messages from the PowerPlant and for each message I get, I evolve the state of my Actor.
There is another upstream Microservice that actually reads the state of the state machine for each actor. So far so good!
I now want to have redundancy built in. This means that I want multiple instances of my Microservice that contains the StateMachine to be run. The problem now is how will the upstream systems see a consistent state for a single PowerPlant? The upstream systems gets state messages from the time they ask to the Microservice via a HTTP endpoint the current state of the PowerPlant.
I see that there are several possibilities with which I could replicate the Akka actor that contains the StateMachine, but I'm not sure if any of those solutions might work! One such approach is here
https://doc.akka.io/docs/akka/2.5.5/scala/distributed-data.html
Any other suggestions?
EDIT: I have a full running application here - https://github.com/joesan/plant-simulator/ This application contains the StateMachine that I was talking about! It you navigate to the https://github.com/joesan/plant-simulator/tree/master/app/com/inland24/plantsim/services/simulator/onOffType and you will find there an Actor and the corresponding StateMachine. The StateMachine is evolved by the messages the Actor receives!
One classic solution is to use Akka Cluster Sharding coupled with Akka Persistence.
Sharding means that each PowerPlant exists only once in your cluster (no data sync issues), but that if the server it resides on goes down (exits the cluster), the actor is re-created somewhere else. Akka Persistence then makes sure your restore the actor state when it’s recreated.
Documentation:
https://doc.akka.io/docs/akka/current/cluster-sharding.html
https://doc.akka.io/docs/akka/current/persistence.html
Related
I'm struggling with proper design of communication between 2 separate akka microservices/bounded contexts in akka cluster.
Lets say we have 2 microservices in cluster per node.
Both microservices are based on akka.
It's clear for me that communication within particular bounded context will be handled by sending messages from actor to actor or from actor on node1 to actor on node2 (if necessary)
Q: but is it OK to use similar communication between separate akka application? e.g. boundedContext1.actor --message--> boundedContext2.actor
or it should be done via much clearer boundaries: In bc1 raise an event - publish to broker and read the event in bc2?
//Edit currently we've implemented a service registry an we're publishing events to service registry via Akka streams.
I think there is no universal answer here.
It is like if your BCs are simple enough you can hold the BCs in one application and even in one project/library with very weak boundaries i.e. just placing them into separate namespaces and providing an API for other BCs.
But if your BCs become more complex, more independent and require its own deployment cycle then it is definitely better to construct more strong boundaries and separate microservices that communicate through message broker.
So, my answer is you should just "feel" the right way according to your particular needs. If you don't "feel" it then follow KISS principle and start with an easier way i.e. using built-in akka communication system. And if in the future your BCs will become more complex then you will have to refactor them. But this decision will be justified and it will not be an unnecessary overhead.
As I understand with akka-cluster, frontend node receives the request and sends the job to one of the backend nodes for executing it. I want to find out for debugging purpose how to know which of the backend node is executing the job?
Does akka also provides some UI where one can look for current job executions happening on different backends?
There is nothing in Akka Cluster that is specifically about work scheduling or frontend and backend nodes, that would just be one application out of many that you can possibly build on top of Akka Cluster and as such, if you would want an UI of some kind you would build that as well for your application.
I am evaluating whether to use Akka and akka persistent as key toolkit of certain project, in which there is a complex background running process (might be triggered by Quartz at a fixed time per day).
The back-ground running process will communicate with many different external services via HTTP communication, will generates many encrypted files locally, and transfer them via SFTP.
From business perspective:
The service is mission critical, roughly it will charge N million users' money from their bank cards automatically and help them purchase some fund product.
From technical perspective:
Each of the external service might not be available with whatever reason, such as network issues, the external service might running out of their resources(i.e. jdbc connections).
Our service might be killed, restarted, re-deployed due to urgent reason or crashed with some unexpected errors.
Once the process was restarted with an incomplete job, then it needs to gracefully complete them with different strategies, such as redo, confirm external system business state, and resume from certain check point.
I was reading from official AkkaScala.PDF, and some youtube conference videos, all of them were mentioning, actor's state can be restored by replaying the events from journal after JVM crash.
But it must be a stupid question, since i did not find it was being discussed:
Imagine there were 1000 persistent actors living in the service, and the service's JVM crashed and restarted, who should be in charge of triggering re-create those 1000 persistent actors in the newly created actor system in both single process mode and clustered mode? And how? Or what articles should I read first?
You should read basics of Akka Persistence and Akka Persistence Query. But probably, first thing that comes to my mind is to use Akka Persistence Query AllPersistenceIdsQuery or CurrentPersistenceIdsQuery. It will give you all persistence id's which you can use to reignite your persistent actors. Persistent actors by specific persistent id will replay all events from event store journal. You can take snapshots to speed up recovery. Your event store will probably be some kind of database (e.g. Cassandra). Considering that your persistent actor has specific mutable state, it will be brought back to its last state after the recovery. Recovery might take some time.
I am looking to build a system that is able to process a stream of requests that needs a long processing time say 5 min each. My goal is to speed up request processing with minimal resource footprint which at times can be a burst of messages.
I can use something like a service bus to queue the request and have multiple process (a.k.a Actors in akka) that can subscribe for a message and start processing. Also can have a watchdog that looks at the queue length in the service bus and create more actors/ actor systems or stop a few.
if I want to do the same in the Actor system like Akka.net how can this be done. Say something like this:
I may want to spin up/stop new Remote Actor systems based on my request queue length
Send the message to any one of the available actor who can start processing without having to check who has the bandwidth to process on the sender side.
Messages should not be lost, and if the actor fails, it should be passed to next available actor.
can this be done with the Akka.net or this is not a valid use case for the actor system. Can some one please share some thoughts or point me to resources where I can get more details.
I may want to spin up/stop new Remote Actor systems based on my request queue length
This is not supported out of the box by Akka.Cluster. You would have to build something custom for it.
However Akka .NET has pool routers which are able to resize automatically according to configurable parameters. You may be able to build something around them.
Send the message to any one of the available actor who can start processing without having to check who has the bandwidth to process on the sender side.
If you look at Akka .NET Routers, there are various strategies that can be used to assign work. SmallestMailbox is probably the closest to what you're after.
Messages should not be lost, and if the actor fails, it should be passed to next available actor.
Akka .NET supports At Least Once Delivery. Read more about it in the docs or at the Petabridge blog.
While you may achieve some of your goals with Akka cluster, I wouldn't advise that. From your requirements it clearly states that your concerns are oriented about:
Reliable message delivery (where service buses and message queues are better option). There are a lot of solutions here, depending on your needs i.e. MassTransit, NServiceBus or queues (RabbitMQ).
Scaling workers (which is infrastructure problem and it's not solved by actor frameworks themselves). From what you've said, you don't even even need a cluster.
You could use akka for building a message processing logic, like workers. But as I said, you don't need it if your goal is to replace existing service bus.
The Akka doc talks about a variety of seemingly inter-related Akka technologies without distinguishing much between them:
Akka Networking
Akka Remoting
Akka Clustering
The Akka ZeroMQ module
My understanding is that "Akka Networking" is simply a module/lib that gives Akka the ability to speak to remote actor systems over TCP. Akka Remoting is another module/lib (not contained in the core Akka JAR) that gives Akka the use of Gossip protocols. And Akka Clustering is yet another module/lib that then uses these Gossip protocols to allow remote actor systems to cluster together and sharestate changes in a viral/"service discovery"-esque manner. And my understanding of Akka ZeroMQ is that it accomplishes the same thing as Akka Clustering, except using ZeroMQ as the basis of the network connections and protocols (instead of Gossip).
So first, if my understanding of these different modules/libs are incorrect, please begin by correcting me!
Assuming I'm more or less on target here, then my main concern is that I might have Remote Actor System 1 (RAS1) using Akka Clustering (and hence Gossip) trying to communicate with Remote Actor System 2 (RAS2) which uses Akka ZeroMQ. In this case, we're using two completely different clustering technologies and protocols, so does this mean these two remote systems can't speak to each other, or does special care need to be taken so that they are compatible with each other?
Akka Remoting is what allows for one actor to speak to another actor on a different machine. For Akka Remoting to work you need to know the specific IP address (or hostname), ActorSystem name, and Actor path of the actor you want to talk to. The ActorSystem name can be different in the 2 machines.
Akka Clustering takes away the problem of having to know the specific machine you are talking to (via Cluster-aware Routing or via a Receptionist that listens for machines joining or leaving the cluster). Cluster-aware Routing also allows for things like having a minimum of X instances of an actor running on any machine in the cluster. Akka Clustering uses the Gossip protocol to maintain the list of cluster members. A cluster-enabled application must know the address of at least one host which must always be running to be able to join the cluster. There might be 2, 3 or more, but the idea is that at least one of them must always be up. Akka Clustering is built on top of Akka Remoting.
Although I haven't worked with Akka ZeroMQ, I assume it works similarly to Akka AMQP. I see it more as an alternative to Remoting, in the sense that it enables actors on different machines to talk to each other, with the advantage that none of the actors need to know any specifics about any other machines where actors are running. However, as with Remoting, you need to manually create the actors that receive the messages, whereas with Clustering the cluster takes care of it (as long as you've configured your Routers correctly).
Regarding your last question. The easiest way I can think about for a Cluster to talk to Akka ZeroMQ would be to have one (or several?) actors in the Cluster that talk to ZeroMQ actors (i.e., you can actually mix and match). Have an actor inside the Cluster that listens on the queue, and have another one that publishes the message to the queue. Sort of an Adapter pattern.