Why so few mountain bikers with spandex?
As already said, aerodynamics are less important to MTB's, but otherwise its largely convention and fashion that dictate what people wear.

A vast majority of MTB'r are not wearing basic shorts - they are usually wearing shorts made for riding, including padding just like Lycra road shorts, flat seams and materials designed to withstand the rigour of riding. Many MTB shorts consist of removable lycra under shorts with the baggy outer designed for cycling.

For me personally, I wear baggies MTBing because we often spend time "in public" - around cafes or pubs after the ride, socialising and I feel like a fashion mistake (and I am not fashion concious) walking into a cafe in lycra. I will wear lycra if doing long distances, or I know I won't be more than 10 feet from the bike for the whole time I am out, but feel more comfortable wearing shorts.