How to automate the process of patching Jenkins Slave instances in AWS I've been working on an automated patching solution for some infrastructure in one of our AWS accounts. This AWS happens to host the Jenkins instance we use for CI/CD actions, among other things. 

Our instance runs off AWS EC2 instances and are persistent - basically they live long enough that they would be out of compliance if we never patched them. 

I have patched Jenkins slaves manually in the past, where the workflow looked something like this:


Log into Jenkins instance
Make sure there are no running jobs on the target node
Mark the target node as offline
Patch the target node
Log into Jenkins instance
Mark the node as online again


This process involves interacting with the Jenkins instance itself. Since I want this solution to be automated, I don't want to have to manually log in and facilitate this process before and after patching. Right now, I have the ability to execute arbitrary bash code from the target node locally and I've been working on a way of using the target node to mark itself offline in some way, and right now my best bet looks like using the environment variables that Jenkins sets while it is running a job to determine whether or not the node is busy. 

That will solve the problem of the making sure the node isn't busy, but I have no idea how I will prevent it from accepting new jobs while automation patches the machine. I don't want to use API calls because I believe that would require me to put credentials on all of the Jenkins machines, which I am adverse to doing (Secrets management is a pain). Another possible method could be to prevent the Jenkins master from assigning the slave more tasks by shutting off the slave's SSH, but that sounds like a terrible idea.

Anyone have any ideas as to how I could mark Jenkins slaves offline and online, with my only real tool being local bash execution on the slave itself?
If you use the Self-Organizing Swarm Plug-in to connect your build agents to your master, you can spin agents up/down without having to make any changes on your master. Agents will automatically register themselves with the master. You will have to supply credentials to the agents to self-register. I would suggest using SSM Parameter Store to store them and then retrieve them in userdata.

Your agents can then be completely ephemeral, and updates performed in userdata or by updating the AMI.

See this Terraform module for a more detailed example of implementing this.