Why/how did flat bars take over?
Flat bars are easier to learn on than drop bars (unless the latter have interrupter brake levers). At this stage low speed manoeuvrability matters more than efficiency. Some people with small hands apparently have trouble braking firmly from the hoods and getting into the drops is tricky for a novice (I dismiss starting in the drops as I've never got on with it). Then people like what they're familiar with if they lack a pressing reason to switch. Drop bars - as you said - need to be right to be comfortable, and the riders who get even the most basic bike shop fitting (not good enough for road bikes really) are in the minority.

Some people (e.g. me) like the extra height of flat bars for seeing and being seen over cars. 

Cruiser bars (apart from being a horrible angle for your wrists if the top tube is on the short side and/or you have broad shoulders) lead to odd handling on some frame geometries, and make it hard to stand up on the pedals. So they don't work for MTB use. And MTBs have sold well since they went mass market (including in BSO form as evidenced in a recent question here). This may have something to do with the forgiving geometry - including the hands position. Don't underestimate the value of a bike that doesn't need much fitting (just saddle height) to be adequate.