Can I shut off the cold water supply to an oil furnace while away on vacation?
It should not be a problem. Your boiler should have a cut-off switch so if, for example, a baseboard springs a leak then the pressure drop will turn off the boiler. If your boiler is old enough to not have such a switch, then turning off the water is probably a bad idea.