2 different batteries in single circuit
Russell has covered the ground connection. I would just like to mention the option of a switching regulator in a case like this.

Instead of using the 3 x 1.5V batteries for your 4.5V supply, you could consider using a buck converter. This type of switching regulator is more efficient (typically >80%) than a simple linear regulator (which will be less than 50% efficient with such a drop).

Here is an example circuit using a linear regulator (LT1085) and a switching regulator (LT1933)
Both have the same input voltage (12V) and are set to ~5V output and 50mA current draw:



Here is the simulation. The linear regulator traces are in the top graph, and the switching regulators in the bottom. They show the power from the supply, the power consumed by the load, and the efficiency calculated from these (Load_Power/Input_Power). We can see the linear regulator is only around 36% efficient, compared to the 82% efficiency of the buck converter (open in a new tab for larger view):