How to prevent actions using Flag (or similar) module? The Goal: I'm trying to setup functionality similar to a library with downloadable files. Only one user can download a file at a time. 

The Solution: Use the Flag module.

Open to suggestions but through my research it seems like people are using the Flag module for this type of functionality. My question is how do I actually get this to work? I'm using a file field to add a file to a node. An admin comes along, wants to "checkout" that file, and they click the checkout flag, and then download the file. Now no other admin can download that file.

Once the original admin comes back, they can just click the flag again, setting it to open. (I know, they aren't really returning a digital file.) They just need to simulate returning it by basically just telling the system someone else can get this file.

Anyone set up something similar to this or just have an idea of how I would do this? Thanks for any suggestions and help.
I haven't done this, but if you want a library check in/check out system this is the approach I would try.  Create a flag (here I'll call it Check out) that allows users to "check out" files.  Make sure this is a global flag.  Name the unflag link "Check in" or something similar.  (You could also automatically have files checked back in after a certain interval with Rules Scheduler.)

Then, I would create views so that users cannot see the nodes that are checked out.  To actually enforce security, I would also add a rule that automatically redirects users away from the page of any node that they did not flag that has been flagged (this probably will cause a performance hit and may not be the most secure way but will probably work for general cases).