Jenkins managing a pool of servers across restarts I have a pool of servers to test a provisioning process my team owns.  One server is needed per branch or PR.  The process requires reboots.  Once the process is run, we run some tests.  A Jenkins pipeline automates the provisioning process and test.

So my problem is how to manage the pool of servers so that 


Any available pool server can be selected for a build
A server claimed for a build is unavailable for other builds
Upon successful build, the server is made available again
Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)


I can't just use the executor on the slave node because we reboot the server and that would kill the job.

My current solution is:


Pool servers are setup as Jenkins slaves with one executor each and a common label
A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves slave.jar to protect against Jenkins restarting the assigned slave (which it annoyingly will do)
The next pipeline stage uses node.getComputer().disconnect() to disable the slave--it seems this must run on master so must be in a different stage
A post { success { ... } } uses ssh to move slave.jar back into place on the server and node.getComputer().connect(true) to allow it to be picked again


This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 &amp; 3.  When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave.  When this happens, things break badly.

Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?

Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run?  Plugins I've missed, etc.

I've examined the few results google returns for the slave disconnect problem and the most promising one was https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw -- the suggestion there failed for me because OfflineCause.UserCause class could not be resolved in my pipeline.  I only know enough Java and Groovy to be dangerous!
I found the missing piece in the Lockable resource plugin.

lock(label: 'server-pool', quantity: 1) {
   ...
}


Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.