Does battery voltage actually get lower when connected to a load, or does it just appear to do so?
Yes, it does get lower.

The effect you see is called internal resistance:


  A practical electrical power source which is a linear electric circuit may &lt;...> be represented as an ideal voltage source in series with an impedance. This impedance is termed the internal resistance of the source.


Put simply, a battery is not an ideal voltage source. A typical battery (i. e. non-ideal voltage source) will look like this:



What you are measuring is voltage between terminals A and B. According to Ohm's Law:

$$ U_{AB} = E * \frac{R}{R+r} $$


When there is no circuit, you can imagine your voltmeter's internal series resistance \$ R_{volt} \$ taking the role of \$ R \$. However, \$ R_{volt} \$ is usually so large (tens or hundreds of megaohms) compared to \$ r \$ (usually fractions of an ohm) that \$ \frac{R_{volt}}{R_{volt}+r} \$ tends to 1, hence measured open-circuit voltage tends to the battery's internal (true) voltage \$ E \$.
When there is a closed circuit with equivalent series resistance of \$ R \$, you will be able to see that the measured voltage \$ U_{AB} \$ drops proportionally to \$ R \$, in accordance with the above formula.


So, the voltage drop is real — the measured voltage is what your load gets. The more current it draws from the battery, the lower is voltage it gets.