Why do front and rear wheel hubs have different sized ball bearings?
First of all, the standard shimano cup and cone bearing size is not necessary, take their own XTR FH-M975, this had 3/16th bearings in a rear hub.

One thing I considered just now was that an MTB will sometimes be on only one of the hubs, so it is unlikely that there are issues to do with damaging either bearing size in this way.

What is left is the typical average load distribution; bearing size and load are both factors in bearing life, as such my overall conclusion is actually that this is a bearing life issue more than anything else, they're simply attempting to obtain the same life from the front and rear. Even as bearings improve this provides a valid reason to keep the size disparity to increase rear bearing life.

http://www.skf.com/us/products/bearings-units-housings/principles/bearing-selection-process/bearing-size/size-selection-based-on-rating-life/index.html

Outside of cup and cone one can take the Hope Pro2 Evo for example, the front takes 20x32x7 while the rear takes 17x30x7, the bearings in both are the same size, but the rear has less of them. I'm guessing that the front only takes more bearings to increase the cartridge radius to accomodate larger axel standards. It's worth noting there are another 3 bearings in the freewheel of the hub, I'm unsure how this affects things, but then there are also bearings in the shimano freehub. Edit: Anecdotally the rear bearings on my pro2 evo needed replacing sooner than the front.

The load really is significantly different:

Lennard Zinn on measuring rear and front load, and "don't be surprised if it's 70/30"

http://www.velonews.com/2015/01/bikes-and-tech/technical-faq/technical-faq-weight-distribution-compatibility_357312

For otherwise identical bearings the load it can take increases with size.

https://www.bearingworks.com/bearing-sizes/

The above is not ideal as it deals with 6000 series cartridge bearings, where the number of bearings varies as well as their size, but it shows clearly that the the load rating increases with size of bearing (when the width increases).

If you want to get more in depth with regard to anything and everything bearing related then check out and explore starting from

http://www.skf.com/us/products/bearings-units-housings/principles/bearing-selection-process/bearing-size/index.html

http://www.nmbtc.com/bearings/engineering/load-life/

EDIT:
My final thought is that cup and cone are the staple of cheap hubs, any change would presumably increase costs (at least initially) and go against standard expectations for bearing sizes, any increase in cost to a manufacturer is passed on to the consumer, in the case of budget hubs there's really no advantage to the consumer for the increased cost, I mean, they work, they work very well for a very long time when maintained, and surprisingly well for surprisingly long even when they're not! Even if you could show this or that improvement in weight or rolling in an optimised hub, would this actually translate to more profit? I doubt it very much. So manufactures just keep things the way they've been, because there's no advantage to be had for the company by changing it.

Also consider the complaints from people that you messed with bearing sizes, no doubt there will be enough people who will put the wrong bearings into the new hub, who will then proceed complain that your company sucks! There are countless examples of technically sub-optimal solutions everywhere you look, many of them are however optimal for profits.

I strongly believe (but can't prove) that a simple philosophy that "both wheels should last equally long" was the main driving force behind the initial decision.