I have a pool of servers to test a provisioning process my team owns. One server is needed per branch or PR. The process requires reboots. Once the process is run, we run some tests. A Jenkins pipeline automates the provisioning process and test.

So my problem is how to manage the pool of servers so that 

1. Any available pool server can be selected for a build
2. A server claimed for a build is unavailable for other builds
3. Upon successful build, the server is made available again
4. Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)

I can't just use the executor on the slave node because we reboot the server and that would kill the job.

My current solution is:

1. Pool servers are setup as Jenkins slaves with one executor each and a common label
2. A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves `slave.jar` to protect against Jenkins restarting the assigned slave (which it annoyingly will do)
3. The next pipeline stage uses `node.getComputer().disconnect()` to disable the slave--it seems this must run on master so must be in a different stage
4. A `post { success { ... } }` uses ssh to move `slave.jar` back into place on the server and `node.getComputer().connect(true)` to allow it to be picked again

This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 & 3. When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave. When this happens, things break badly.

Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?

Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run? Plugins I've missed, etc.

I've examined the few results google returns for the slave disconnect problem and the most promising one was <https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw> -- the suggestion there failed for me because `OfflineCause.UserCause` class could not be resolved in my pipeline. I only know enough Java and Groovy to be dangerous!