Is there any proper way to down a node when this become unreachable in akka cluster. I want to expose an api to down a node when it becomes unreachable but I prefer to find another way or something programmatically. Auto-downing is not an option anymore since I've been having split brain issues.
Can I just down the node in the receive method :
def receive: Receive = {
case MemberUp(member) =>
case UnreachableMember(member) => {
Cluster(context.system).down(member.address)
}
If the node has crashed, downing it on any other node is fine.
But you must do it as a reaction to seeing it unreachable though, that will cause split brain if it is a network partition rather than a node crashing the unreachable each side of the partition will see the rest of the cluster as unreachable and down all those, and you will end up with two clusters thinking that they have downed the other side.
This is the exact reason why we have removed auto downing in Akka 2.6
If you want automatic removal of nodes with split brain resolution you will need to use/implement a downing provider that handles that for you.
Implementing a downing provider yourself will require quite a bit of effort and thinking it through. For more background on the problem you can read the docs of the commercial Lightbend SBR here: https://doc.akka.io/docs/akka-enhancements/current/split-brain-resolver.html
Related
I'm trying to find a workaround to the following limitation: When starting an Akka Cluster from scratch, one has to make sure that the first seed node is started. It's a problem to me, because if I have an emergency to restart all my system from scratch, who knows if the one machine everything relies on will be up and running properly? And I might not have the luxury to take time changing the system configuration. Hence my attempt to create the cluster manually, without relying on a static seed node list.
Now it's easy for me to have all Akka systems registering themselves somewhere (e.g. a network filesystem, by touching a file periodically). Therefore when starting up a new system could
Look up the list of all systems that are supposedly alive (i.e. who touched the file system recently).
a. If there is none, then the new system joins itself, i.e. starts the cluster alone. b. Otherwise it tries to join the cluster with Cluster(system).joinSeedNodes using all the other supposedly alive systems as seeds.
If 2. b. doesn't succeed in reasonable time, the new system tries again, starting from 1. (looking up again the list of supposedly alive systems, as it might have changed in the meantime; in particular all other systems might have died and we'd ultimately fall into 2. a.).
I'm unsure how to implement 3.: How do I know whether joining has succeeded or failed? (Need to subscribe to cluster events?) And is it possible in case of failure to call Cluster(system).joinSeedNodes again? The official documentation is not very explicit on this point and I'm not 100% how to interpret the following in my case (can I do several attempts, using different seeds?):
An actor system can only join a cluster once. Additional attempts will
be ignored. When it has successfully joined it must be restarted to be
able to join another cluster or to join the same cluster again.
Finally, let me precise that I'm building a small cluster (it's just 10 systems for the moment and it won't grow very big) and it has to be restarted from scratch now and then (I cannot assume the cluster will be alive forever).
Thx
I'm answering my own question to let people know how I sorted out my issues in the end. Michal Borowiecki's answer mentioned the ConstructR project and I built my answer on their code.
How do I know whether joining has succeeded or failed? After issuing Cluster(system).joinSeedNodes I subscribe to cluster events and start a timeout:
private case object JoinTimeout
...
Cluster(context.system).subscribe(self, InitialStateAsEvents, classOf[MemberUp], classOf[MemberLeft])
system.scheduler.scheduleOnce(15.seconds, self, JoinTimeout)
The receive is:
val address = Cluster(system).selfAddress
...
case MemberUp(member) if member.address == address =>
// Hooray, I joined the cluster!
case JoinTimeout =>
// Oops, couldn't join
system.terminate()
Is it possible in case of failure to call Cluster(system).joinSeedNodes again? Maybe, maybe not. But actually I simply terminate the actor system if joining didn't succeed and restart it for another try (so it's a "let it crash" pattern at the actor system level).
You don't need seed-nodes. You need seed nodes if you want the cluster to auto-start up.
You can start your individual application and then have them "manually" join the cluster at any point in time. For example, if you have http enabled, you can use the akka-management library (or implement a subset of it yourself, they are all basic cluster library functions just nicely wrapped).
I strongly discourage the touch approach. How do you sync on the touch reading / writing between nodes? What if someone reads a transient state (while someone else is writing it) ?
I'd say either go full auto (with multiple seed-nodes), or go full "manual" and have another system be in charge of managing the clusterization of your nodes. By that I mean you start them up individually, and they join the cluster only when ordered to do so by the external supervisor (also very helpful to manage split-brains).
We've started using Constructr extension instead of the static list of seed-nodes:
https://github.com/hseeberger/constructr
This doesn't have the limitation of a statically-configured 1st seed-node having to be up after a full cluster restart.
Instead, it relies on a highly-available lookup service. Constructr supports etcd natively and there are extensions for (at least) zookeeper and consul available. Since we already have a zookeeper cluster for kafka, we went for zookeeper:
https://github.com/typesafehub/constructr-zookeeper
In this moment I have this actor session management implementation running in only one node:
1) I have a SessionManager actor that handles all sessions
2) The SessionManagerActor receives two messages: CreateSesion(id) and ValidateSesion(id)
3) When the SessionManagerActor receives CreateSesion(id) message, it creates a SessionActor using actorFor method like so:
context.actorOf(Props(new SesionActor(expirationTime)), id)
4) When the SessionManagerActor receives ValidateSesion(id) message it looks for an existing SessionActor and evaluates if exists using resolveOne method like so:
context.actorSelection("akka://system/user/sessionManager/" + id).resolveOne()
With that logic works nice but I need to implement the same behavior in multiple nodes (cluster)
My question is, which method is recommended to implement that session management behavior so that it works in one or múltiple nodes?
I've read akka documentation and it provides akka-remote, akka-cluster, akka-cluster-sharding, akka-cluster-singleton, akka-distributed-publish-subscribe-cluster but I'm not sure about which one is the appropriate and the simplest way to do it. (Note that SessionActors are stateless and I need to locate them anywhere in the cluster.)
Since you have a protocol where you validate whether a session already exists or not and have a time-to-live on the session, this is technically not completely stateless. You probably would not, for example, want to lose existing sessions and spin them up again arbitrarily, and you probably don't want to have multiple sessions created per id.
Therefore, I would look at the cluster sharding mechanism, possibly in combination with akka-persistence to persist the expiration state of the session.
This will give you a fault tolerant set up with rebalancing when nodes go down or new nodes come up.
The activator template akka cluster sharding scala may be helpful for example code.
EDIT: One important thing I forgot mentioning: the actor creation described below depends on the data - sometimes few processing actors are required, and sometimes many.
One component I'm working on needs to create a number of actors (possibly, round-robin routed ones), that each get a rather large amount of messages to process. Each of those actors belongs to a "processing batch" which has a the same initialization parameters.
When I'm running this on the production machine with many messages, I quickly get a number of actor creation timeouts. I'm creating the actors directly with ActorSystem.actorOf().
What's surprising me though is that all in all there aren't that many actors being created I'd think (8 "processing sinks" with 5 round-robin routed actors would be 40 actors, which doesn't seem very much).
I'm shutting down the actors once they're not needed anymore by having another actor (which counts the amount of successes and failures that it gets via the "processing" actors) send them a PoisonPill so I'd think that they are all shut down correctly.
Am I perhaps doing something wrong here in the way that I am creating those actors, e.g. should I perhaps create them differently? Or would an appropriate strategy be to wait for some of the batches to be done before creating new actors?
Since you did not specify which version you are using I’m assuming that you will be interested in reading this:
http://doc.akka.io/docs/akka/2.0.3/scala/actors.html#Creating_Actors_with_default_constructor (especially the warning)
Besides the technical argument, creating your actors at top level is not a good design, missing out on the fault handling benefits.
Can someone describe in very simple terms how you would scale up a service (lets assume the service is very simple and is the function X() ).
To make this scalable would you just fire off a new node (upto a maximum depending on your hardware) for each client who wants to run X?
So if I had four hardware boxes, I may fire up to four nodes to run service X(), on the 5th client request I would just run X() on the first node, the 6th client on the second node etc?
Following on from this, I know how to spawn processes locally, but how would you get both the 1st and 5th clients to use the same Node 1- would it be by spawning a process remotely on the Node for the Client each time?
Any simple examples are most welcome!
This depends very much on what X is. If X is fully independent, for instance x() -> 37. then you don't even need to connect your nodes. Simply place some standard Load Balancer in front of your system (HAProxy, Varnish, etc) and then forget about any kind of distributed communication. In fact, there is no need to use Erlang for that. Replace Erlang with some other language of your choice. It is equally good.
Where Erlang shines is when several X functions have dependencies on each others result and when the X might live on another physical machine. In that case Erlang can communicate with the other X seamlessly, even if it lives on a different node.
If you want to implement a round-robin scheme in Erlang, the easiest way is to have a single point of entry and then let it forward the requests out to multiple nodes. But this is bad if there is a pattern where a certain node ends up with all the long-running processes. You need to build a mechanism of feedback so you know how to weight the round-robin queue.
I am coding a workload scheduler. I would like my piece of software to be a peer-to-peer scheduler, ie. a node only knows some neighbours (other nodes) and use them to reach other nodes.
Each node would have its own weighted-routing table to send messages to other peers (basically based on the number of hops), ie. "I want the master to give me my schedule" or "is resource A available on node B ?" : which neighbor is the closest to my target ?
For instance I have written my own routing protocol using XML-RPC (xmlrpc-c) and std::multimaps / std::maps.
I am thinking of using ZeroMQ to optimze my data streams :
queueing can reduce the network load between peers ;
subscriptions can be used to publish upgrades.
As a consequence :
I would need to open as many sockets as I would create new types of connections ;
Each node would need to be a client, a server, a publisher, a subscriber, a broker and a directory ;
I am not sure that my "peer-to-peer architecture" is compatible with the main purpose of ZeroMQ.
Do you think that ZeroMQ can be a helpful concept to use ?
It would be helpful to know exactly what you mean by "routing protocol".
That sounds like you mean the business logic of routing to a particular peer.
Knowing more fully what you're looking to achieve with ZeroMQ would also be helpful.
Have you read the ZeroMQ Guide?
ZeroMQ is a pretty different beast and without spending some time to play with it, you'll
likely find yourself confused. As a bonus, reading the guide will also help you answer
this question for yourself, since you know your requirements better.
ZeroMQ was designed to build robust distributed and multi-threaded applications. Since distributed applications can often take the form of "peer-to-peer", ZeroMQ could indeed be a good fit for your needs.