Why won't my furnace turn off?
In my experience the old mercury trip thermostats with the bi-metallic strip are wildly inaccurate. My experience was similar to yours, I had to crank the thermostat down as low as it could go to get the furnace to turn off and crank it high to get it to turn on.

I'd recommend going to the local hardware store and picking out an inexpensive digital thermostat. Here is a link to a good example. It doesn't have a ton of features but it will get you the accuracy you're looking for. They're pretty easy to install, only a few wires to hookup (2 for your furnace and 3 for central AC if I remember correctly).

I've installed a similar thermostat in every place I've lived in if a mercury trip thermometer was in place ... even rented apartments.

There may be a problem with your furnace but this is a good first step to take and even if it doesn't solve the problem it's a good investment for your place. Easy to read, easy to change and very convenient.