Is it OK to turn off power to a fuel furnace?
Yes, it is fine to turn off the power, and you should. The wires going to the thermostat are low-voltage (typically 24V AC), but connecting the wrong ones can still potentially damage the control circuitry. 

Most furnaces made in the last 10-20 years use electronic ignition, so there is no pilot light to worry about (if that's what you are worried about). Even with a pilot light, I don't believe it would be an issue, but at worst case you'd have to re-light the pilot.