Problems running Arduino project off a 9V battery
As soon as I connect up the battery, I loose the Serial Monitor connection (even though the USB cable is still connected). So I can no longer see what's going on inside the Arduino for debugging purposes. Why? How can I continue to see the Serial Monitor? Is there a way to see the serial output from somewhere other than the USB jack?


Probably because although the voltage is high enough to cut off the power from the USB port it's not got enough current to run the main MCU properly.


  I can tell by the lights on the board and by the behavior of the outputs that the software is malfunctioning. But I can't tell exactly why (because of #1). Why would changing the power source cause this?


Because 9V batteries can't provide enough current.


  I know that the VIN pin needs 7-12 volts. The battery I plugged in is a new 9V battery. But what happens as it wears down? Does it still push 9 volts? Or does the voltage drop? Can I tell my Arduino to stop functioning if the voltage drops below acceptable levels? Because ceasing altogether is better than malfunctioning.


No. The voltage drops rapidly to about 8V then down and down and down until it's below what the regulator can regulate. You can see this curve in any battery datasheet.


  I have read that I need to ensure that the current from the battery remains around 500mA. But how can I measure that in hardware/software? This ties back into question #3.


That's nonsense. The current demands will be whatever the circuit needs at any moment in time. There will be an upper limit to that demand (defined by your circuit and what you are doing at any one time - the peak current draw) and your power source has to be able to provide at least that amount of current.  If you try to draw more current than the battery can handle then the output voltage will drop to a level where it can no longer power the circuit properly. It is at this point you get "brownouts" and the board resets randomly and strange unpredictable things happen.

In short: never use a 9V battery for an Arduino project. It's fine for vert low power operations, which take careful design, but for general use stay away from them. Also the Arduino's built-in regulator is a very inefficient and will waste about 25%-40% of a 9V batteries power before you even begin. Instead you should use a more efficient switching regulator to provide 5V directly to the 5V pin (or USB port) from suitable higher capacity batteries (such as AA cells).



As a comparison here's some typical battery discharge graphs comparing alkeline AA to PP3 (9V). These are both Duracell Ultra.



As you can see the PP3 drops its voltage very rapidly even under a light load. Under a "heavy" load (what would be considered a very light load to an AA) it tumbles away to next to nothing in no time at all.