Why is the voltage measurement of this circuit different when the switch is on?
First of all, well done for doing the practical experiment. It's good to see people trying things and wanting to discover the answers to their observations.

It may be easier to visualise this with a circuit. As it has been pointed out in a previous answer, batteries have internal resistance. So if you add a load on to it, you create a voltage divider. As the battery becomes more and more drained, the internal resistance increases, which stops it being able to power the load.







simulate this circuit &ndash; Schematic created using CircuitLab

Look at the above schematic. We all know Ohm's Law, which is V = I*R. If we give values to the resistors we can calculate how much the drop will be when you close the switch. If we say the battery internal resistance is 0.5 Ohms, and the load is 100 Ohms we can find out the current in the circuit. To do this, we re-arrange the Ohm's Law equation for I: I = V/R = 9/(100+0.5) = 0.0896 A, or 89.6mA. Using Ohm's Law again (you will find that this is probably the most useful equation you will come across in electronics!), we can find the voltage drop over the battery resistance:

Remember: V = I*R = 0.0896*0.5 = 0.0448V. Take this away from the 9V the battery first supplied and you get the voltage you will be measuring when the switch is closed: 9-0.0448 = 8.95V.

Using this knowledge, if you wanted to further your experiments, why not get a known resistor as a load, and power it using different batteries. With your multimeter, you can measure the current and the voltage, which will give you all the numbers you need to calculate the internal resistances of your batteries.

As Marcus Muller points out, temperature can have an effect on this too, so why not do a before and after test, calculate your internal resistance before and after putting them in a fridge/freezer, and see how much it changes. Try it on a few different types of batteries too... There are a lot of cool experiments you can do which will increase your understanding of what is going on, and improve your skills at solving circuits.

Keep it up and good luck!