What is this thing with wires found on wall inside closet, and how do I remove it?
Those certainly looks like doorbell electromagnets, so my guess is you are correct in assuming it is a doorbell chime unit. Maybe a previous owner removed the door button. 

Doorbell chime with electromagnets visible, for comparison:




  My idea at the moment is to shut off the main breaker (since I don't
  know which one this is on), unscrew the wires, put in a box, cap
  wires, and when I drywall put a plate over it for later access if
  needed.


Your proposed solution is correct. Because it is live, you must leave it accessible. 

As you work more on your remodel, try to find where the wires originate. That location will be the transformer for your doorbell. This is where you should disconnect your low voltage wires (the ones going to the doorbell), remove your transformer, and cap your 120V wires. Alternatively, you could simply leave your transformer connected to your 120V and just disconnect the low voltage side going to the doorbell.

Either way, you won't have to leave a box &amp; blank plate in your newly remodeled area. Here is an example of what your transformer setup might look like: