Is there a way to start slurmctld daemon with the execution nodes off, but making it to belive that he has requested the suspend for these nodes (e.g. like if it had called the SuspendProgram)?
I am setting up a virtual cluster, so the SuspendProgram and ResumeProgram do terminate and instanciate virtual machines. In this way I could power on only the master node, and he would fire up nodes only when requested.
The problem is that for the moment, when I start slurmctld I need the nodes to get up, tell him that they exits, and wait that he shut them down. This adds unwanted costs, because I need to poweron all the "supposed" instances.
I would like to instanciate the master, the one running slurmctld, and let him think that the nodes are idle~ like after SuspendProgram.
Cheers
What you can try is set the nodes to state POWER_DOWN in slurm.conf so that at startup, slurmctld will see those nodes as powered down by SuspendProgram
NodeName=... Sockets=... CoresPerSocket... [etc] State=POWER_DOWN
Related
Occasionally my instances get into a corrupted state (especially since min-instance=1). I would like to restart one manually. Is this possible?
I know I can go through the console to create a new version, but this messes up my Terraform state. I would like to keep with the current version and just cycle the instance, a classic IT procedure called "Turning it on and off again" to fix my short term issue while I figure out the larger issue.
No, you can't do it. If you have a routine that can detect a corruption, you can exit the container (the instance stopped and a new one is created). For that, 2 options:
Either you have an internal check that detects automatically the state of the container and exits in case of corruption (works for max-instance >=1 )
Or you can have 2 different endpoints (works only for max-instance=1):
One tell you the state of the container (OK or KO)
In case of KO, you can call an endpoint in your app that stop the instance ( And if your container is public, it's dangerous because anyone can restart your container!)
I'm trying to find a workaround to the following limitation: When starting an Akka Cluster from scratch, one has to make sure that the first seed node is started. It's a problem to me, because if I have an emergency to restart all my system from scratch, who knows if the one machine everything relies on will be up and running properly? And I might not have the luxury to take time changing the system configuration. Hence my attempt to create the cluster manually, without relying on a static seed node list.
Now it's easy for me to have all Akka systems registering themselves somewhere (e.g. a network filesystem, by touching a file periodically). Therefore when starting up a new system could
Look up the list of all systems that are supposedly alive (i.e. who touched the file system recently).
a. If there is none, then the new system joins itself, i.e. starts the cluster alone. b. Otherwise it tries to join the cluster with Cluster(system).joinSeedNodes using all the other supposedly alive systems as seeds.
If 2. b. doesn't succeed in reasonable time, the new system tries again, starting from 1. (looking up again the list of supposedly alive systems, as it might have changed in the meantime; in particular all other systems might have died and we'd ultimately fall into 2. a.).
I'm unsure how to implement 3.: How do I know whether joining has succeeded or failed? (Need to subscribe to cluster events?) And is it possible in case of failure to call Cluster(system).joinSeedNodes again? The official documentation is not very explicit on this point and I'm not 100% how to interpret the following in my case (can I do several attempts, using different seeds?):
An actor system can only join a cluster once. Additional attempts will
be ignored. When it has successfully joined it must be restarted to be
able to join another cluster or to join the same cluster again.
Finally, let me precise that I'm building a small cluster (it's just 10 systems for the moment and it won't grow very big) and it has to be restarted from scratch now and then (I cannot assume the cluster will be alive forever).
Thx
I'm answering my own question to let people know how I sorted out my issues in the end. Michal Borowiecki's answer mentioned the ConstructR project and I built my answer on their code.
How do I know whether joining has succeeded or failed? After issuing Cluster(system).joinSeedNodes I subscribe to cluster events and start a timeout:
private case object JoinTimeout
...
Cluster(context.system).subscribe(self, InitialStateAsEvents, classOf[MemberUp], classOf[MemberLeft])
system.scheduler.scheduleOnce(15.seconds, self, JoinTimeout)
The receive is:
val address = Cluster(system).selfAddress
...
case MemberUp(member) if member.address == address =>
// Hooray, I joined the cluster!
case JoinTimeout =>
// Oops, couldn't join
system.terminate()
Is it possible in case of failure to call Cluster(system).joinSeedNodes again? Maybe, maybe not. But actually I simply terminate the actor system if joining didn't succeed and restart it for another try (so it's a "let it crash" pattern at the actor system level).
You don't need seed-nodes. You need seed nodes if you want the cluster to auto-start up.
You can start your individual application and then have them "manually" join the cluster at any point in time. For example, if you have http enabled, you can use the akka-management library (or implement a subset of it yourself, they are all basic cluster library functions just nicely wrapped).
I strongly discourage the touch approach. How do you sync on the touch reading / writing between nodes? What if someone reads a transient state (while someone else is writing it) ?
I'd say either go full auto (with multiple seed-nodes), or go full "manual" and have another system be in charge of managing the clusterization of your nodes. By that I mean you start them up individually, and they join the cluster only when ordered to do so by the external supervisor (also very helpful to manage split-brains).
We've started using Constructr extension instead of the static list of seed-nodes:
https://github.com/hseeberger/constructr
This doesn't have the limitation of a statically-configured 1st seed-node having to be up after a full cluster restart.
Instead, it relies on a highly-available lookup service. Constructr supports etcd natively and there are extensions for (at least) zookeeper and consul available. Since we already have a zookeeper cluster for kafka, we went for zookeeper:
https://github.com/typesafehub/constructr-zookeeper
I have read the Akka docs on fault tolerance & supervision, and I think I totally get them, with one big exception (no pun intended).
Why would you ever want/need to stop a child actor???
The only clue in the docs is:
Closer to the Erlang way is the strategy to just stop children when they fail and then take corrective action in the supervisor...
But to me, stopping a child is the same as saying "don't execute this code any longer", which to me, is effectively the same as deploying new changes to the code which has that actor removed entirely:
Every Actor plays some critical role in the actor system
To simply stop the actor means that actor currently doesn't have a role any longer, and presumes the system can now somehow (magically) work without it
So again, to me, this is no different than refactoring the code to not even have the actor any more, and then deploying those changes
I'm sure I'm just not seeing the forest through the trees on this one, but I just don't see any use cases where I'd have this big complex actor system, where each actor does critical work and then hands it off to the next critical actor, but then I stop an actor, and magically the whole system keeps on working perfectly.
In short: stopping an actor (to me) is like ripping the transmission out of a moving vehicle. How can this ever be a good/desirable thing?!?
The essence of the "error kernel" pattern is to delegate risky operations and protect essential state, it is common to spawn child-actors for one-off operations, and when that operation is completed and its result send off somewhere else, the child-actor or the parent-actor needs to stop it. (otherwise the child-actor will remain active/leak)
If the child actor is doing a longer process that could be terminated safely, such as video coding, or some kind of file transformation and you have to deploy a new build, in that case a terminate sign would be useful to stop running processes gracefully.
Every Actor plays some critical role in the actor system
This is where you are running into trouble, I can create a child actor to do a job, for example execute a query against a database or maintain the state of a connected user and this is its only purpose.
Once the database query is complete or the user has gracefully disconnected the child actor no longer has any role to play and should be stopped so that it will release any resources it holds.
To simply stop the actor means that actor currently doesn't have a role any >longer, and presumes the system can now somehow (magically) work without it
The system is able to continue because I can create new child actors if/when they are needed.
Can someone describe in very simple terms how you would scale up a service (lets assume the service is very simple and is the function X() ).
To make this scalable would you just fire off a new node (upto a maximum depending on your hardware) for each client who wants to run X?
So if I had four hardware boxes, I may fire up to four nodes to run service X(), on the 5th client request I would just run X() on the first node, the 6th client on the second node etc?
Following on from this, I know how to spawn processes locally, but how would you get both the 1st and 5th clients to use the same Node 1- would it be by spawning a process remotely on the Node for the Client each time?
Any simple examples are most welcome!
This depends very much on what X is. If X is fully independent, for instance x() -> 37. then you don't even need to connect your nodes. Simply place some standard Load Balancer in front of your system (HAProxy, Varnish, etc) and then forget about any kind of distributed communication. In fact, there is no need to use Erlang for that. Replace Erlang with some other language of your choice. It is equally good.
Where Erlang shines is when several X functions have dependencies on each others result and when the X might live on another physical machine. In that case Erlang can communicate with the other X seamlessly, even if it lives on a different node.
If you want to implement a round-robin scheme in Erlang, the easiest way is to have a single point of entry and then let it forward the requests out to multiple nodes. But this is bad if there is a pattern where a certain node ends up with all the long-running processes. You need to build a mechanism of feedback so you know how to weight the round-robin queue.
How to restrict proccess to create new processes?
You could assign the process to a job object. Use SetInformationJobObject with the JOB_OBJECT_LIMIT_ACTIVE_PROCESS flag to limit the number of processes in that job object to one. Do NOT set the JOB_OBJECT_LIMIT_BREAKAWAY_OK (which would allow the process to create processes that were not part of the job object).
The process could still work around that, such as by starting a new process via the task scheduler or WMI. If you're trying to do something like create a sandbox to run code you really don't trust, this won't adequate. If you have a program that you trust, but just want to place a few limits on what it does, this should be more than adequate.
To put that slightly differently, this is equivalent to locking your car. Somebody can break in (or out, in this case), but at least they have to do a bit more than just walk in unhindered.
On Windows, there isn't a way to stop a processing from spawning other processes. Nor is there on any operating system I know of.
The CreateProcess() system call is available to all processes, thus any process can create a child process.
You could run the process in a sandbox which restricts process creation, but the overhead for this is probably more than you want.
Can I ask why you want to do such a thing?
Use NT Job objects
JOBOBJECT_BASIC_LIMIT_INFORMATION can limit the number of active processes, or use JOBOBJECT_ASSOCIATE_COMPLETION_PORT and kill the new process (If you only need to kill a subset of all new processes)