Why is it easier to inverse transform every object in a scene than to transform a camera?
That's actually incorrect. You can transform every ray of your camera if you wish (and numerous implementations do so). There are some advantages and disadvantages to each method (e.g. if your rays are more than your vertices you end up doing more transformations, however you don't incur cache misses by running over all vertices).