Why do only downhill mountain bikers wear full face helmets?
I think there will be a significant subjective element to answering this question, but pressing on:


Full face helmets are relatively heavy
Full face helmets offer less ventilation that standard helmets
Full face helmets are less aerodynamic that standard helmets
Riders don't want to look silly or weird
Full face helmets are not marketed to cross country MTB, road or casual cyclists.


However, given that a few decades ago cycle helmets were rarely worn, but are now common; it seems plausible that in the future lighter, more ventilated helmets will be developed and become generally acceptable for less extreme forms of riding.