I thought I understood the basics of DC circuits and Ohm's law but now I'm confused
I know the formula R=V/I, and therefore that resistance is directly proportional to voltage and inversely proportional to current. 


I would say that, "The resistance defines the relationship of voltage across a resistor to the current through the resistor".


  So if I higher the resistance the voltage increases too right?


Only if the current remains constant. In very many cases the voltage is fixed so increasing the resistance decreases the current as you would expect from \$ I = \frac V R \$.


  Now I'm working on an old RC car I used to play with. The battery pack consists of 6 AA batteries which provide 9 V, but I noticed that there's another output that outputs 4.5 V.


It sounds as though they are tapping off at the battery pack mid-point or they have a voltage regulator stepping down from 9 V to 4.5 V.


  I guess it's being used to power the driver board.


It could be. Many types of logic will run will on about 5 V.


  I tried reading the voltage but I also put a 10 megaohm resistor in the circuit. I was surprised when I saw that the multimeter was reading around 2.2V. Isn't the voltage supposed to be higher because of the resistor?


Bring this to its logical conclusion: with an infinite resistance (no resistor present) you should get infinite voltage? 

No, you have connected 1 M&Omega; in series with a multimeter with what appears to be a 1 M&Omega; input impedance. This has divided the available voltage in two.







simulate this circuit &ndash; Schematic created using CircuitLab

Figure 1. (a) Direct measurement. (b) Voltage measurement through a series resistor.

You can use Figure 1b as a way of measuring the internal resistance of the multimeter.