Why are most minerals so rare?
Mineral rarity is dependent on several factors that include these:


Distribution of elements within the earth


The mass-abundance of the nine most abundant elements in the Earth's crust is approximately: oxygen 46%, silicon 28%, aluminum 8.2%, iron 5.6%, calcium 4.2%, sodium 2.5%, magnesium 2.4%, potassium 2.0%, and titanium 0.61%. Other elements occur at less than 0.15%. (reference)

So minerals composed of elements other than (O, Si, Al, Fe, Ca, Na, Mg, K, and Ti) represent far less than 0.15% chemical composition of the earth.


Redistribution of elements by natural processes.

Consider the top 9 elements (O, Si, Al, Fe, Ca, Na, Mg, K, Ti), notice Chlorine is not in the list but by mass Cl should be in the top 9 for the surface of the earth.  Natural processes will alter the distribution of elements.

Top elements in the oceans (71%) of earth's surface are:
Oxygen, hydrogen, chlorine, sodium, magnesium, sulfur, calcium, potassium, bromine, carbon, and boron.


Atmospheric elements are: Oxygen, Carbon, Hydrogen,Nitrogen ,Calcium, Potassium


Stability (physical and chemical characteristics) of mineral species can affect how common a mineral may be.

Example: Water soluble mineral species - There are at least 90 mineral species that readily dissolve in water.  (reference)

Some minerals are solution readily affected by acidic or alkali solutions.

Some minerals are inherently unstable are near surface conditions.


Some minerals readily oxidize  (native iron)
Some minerals can slowly altered by humidity and oxygen (marcasite)
Some minerals are only stable at high tempature and/or pressure.
(It depends on the crystal structure of the mineral how long the mineral remain at surface conditions (1 atm pressure 25'C)
Diamonds are resilient but alpha-Quartz can rapidly alter to beta-Quartz. 

Mineral formation environments impact mineral distribution.


Minerals that commonly form in the mantle of the earth are naturally going to hard to find at the surface regardless of the chemical position.

Volcanic mineral assemblies require a volcanic activity which is typically limited to plate boundaries and isolated mantle hot spots.


Lastly thermodynamics (Gibbs free energy) can be used to measure the how probability of specific chemical reaction is occur.  

Consider:  Fe + S = FeS2  (pyrite)  verses Fe + S = FeS2  (marcasite)
Free energy state of an environment can be used to explain why pyrite or marcasite will form.  


Some chemical reactions are very improbable due to elements involved with in a specific environment.

Consider:
   Cubic zirconia (CZ) is the cubic crystalline form of zirconium dioxide (ZrO2). It is not found in nature because there are no stable environments for formation where Silicon is absent.   Zircons are formed instead. (zirconium silicate (ZrSiO4).