Why are the mountains predominately grey or dark brown?
As Gary Kindel in his answer correctly said, mountains may consist of either magmatic, metamorphic or sedimentary rocks or all three together.

The greyish to brownish color is thus due to a mixture of all sorts of rocks and minerals that is covered by layers of eroded material of different grain sizes. This way, mountains appear to be of a single color when observed from a distance. This is similar to mixing a couple of watercolors together - the result will either be a greyish or a brown color. However, if we would magnify the watercolor mixture under a microscope, so that we see the individual pigments, we could tell that it isn't actually grey/brown. Same applies to the mountains. From a far, they appear to be grey/brown but when you get closer you can depict all sorts of colors.

The only exceptions to this are mountains, where the material is very well stratified, so that different colors can be seen from a distance.  

To answer your second question: Please note that Olivine, Pyrite and Tourmaline are not rocks but minerals. 

Olivine is mostly abundant in earth's mantle (check out Moho). The chance that a mountain is build solely by a material from the mantle is highly unlikely. What happened to the material that was on top? Secondly, Olivine is not stable at earth's surface according to the Goldich dissolution series. It alters into clay minerals and iron oxides that are easily eroded by e.g. rainwater. Thus, if there was a mountain made of olivine it would have surely been eroded by now. 

Tourmaline is a mineral that forms in pegmatites and is thus far too rare to form an entire mountain. Especially since there is no process that individually picks Tourmalines from nature to build a mountain out of it.

Pyrite; there is still a great debate in the literature about the formation processes of Pyrite. But it is most likely a mineral that forms in a secondary process due to the intrusion of hydrothermal mineral-rich water. Thus, Pyrite is not concentrated in one area and is, like Tourmaline, far too rare to form a mountain.