Do batteries lose voltage as they're used up?
Both effects occur as a battery is drained.  The open circuit voltage goes down and the internal resistance goes up.  Note that open circuit voltage is specifically measuring just the voltage the battery puts out with the internal resistance taken out of the equation.  That is because there is no current thru that resistance, hence no voltage drop across it.  Any decent voltmeter will have at least 10 M&Omega; input resistance, which is so way more than even a dead battery as to not matter.

All that said, different battery chemistries have different characteristics regarding both these parameters as they are drained.  NiCd and NiMH have rather flat discharge curves after a short initial period.  That means the open circuit voltage doesn't drop much for most of the discharge cycle even as the stored energy is getting steadily lower.  These batteries then show a rather steep falloff in voltage as the last 10% or so of energy is drained.  For a NiMH or NiCd therefore, it's tricky to determine a state of charge just from the voltage.

Other chemistries have a more linear discharge curve (voltage as a function of accumulated Coulombs drained at a fixed current).  Old fashioned carbon-zinc cells are more like this.  Usually, there is a significant temperature dependence too, both in terms of voltage and capacity.

Yes, batteries can get complicated.