How can I prevent blender from doing ridiculously large calculations?
Not in the way you are thinking

This is one of those "Computer Science Traps" that sound easy on paper but are actually impossible for computers to solve.  The specific problem at work is the "Halting Problem".  The gist of the issue:


  the Halting problem is the problem of determining, from a description of an arbitrary computer program and an input, whether the program will finish running or continue to run forever....Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist.


There are ways to cheat, however.  Firefox, as mentioned above, cheats by putting time-limits ("timeouts") on operations.  Blender does not have this luxury, because, as you said:


  it takes about an hour to bake the increased resolution.


Which, as annoying as it sounds, is still legitimate.  

The ways around this to make sure you can see the render/bake output, and check in occasionally.  If you suspect that your machine has bitten off more than you can chew, you can usually escape the process by mashing esc