How should one select his or her citations when writing a thesis in STEM?
The general answer to all your questions is to ask your thesis advisor. (S)he will definitely be able to tell you what the customs for theses in your field, and in your university, are.

That being said, I had the impression that you may have some misconceptions about why, and how, we use citations in academic texts. Specifically, you claim that:


  There seems to be no guideline or rule to selecting citations.


I would argue that this is because one does not "select" citations as such. Rather, the literature that your thesis cites should emerge from the text you are writing. You build a scientific argument and then, at the appropriate points in the text, refer back the original text whenever you use a pre-existing piece of knowledge. Sometimes this pre-existing knowledge may be standard enough that a textbook is a good source, but it is not uncommon that you will need to refer to original research papers. 

If those papers have been cited often or rarely so far is not the right question to ask. Instead, if your argument requires that X is true, you find the original paper that proves X. You don't cite a different paper that also somehow uses X because it is more often cited. If you are not aware of any research that actually showed X, you have a different problem - why can you reliably assume X to be the case in your thesis anyway? Your current approach of writing down results and then worrying about what to cite for them seems a little backwards to me.


  All in all I will probably have over 80 citations. Is this too much?


"How many references should I have?" is really the wrong question to ask. You use as many as you need. That being said, if you have very few citations of previous work in a thesis it is usually a bit worrying, as a lack of citations often indicates that a student is not aware of the scientific state of the art or, even worse, of good citation practices. Conversely, citing many previous works is not usually a problem for theses. Of course some upper bounds exist in practice, but 80 citations seems nowhere close to excessive even for a master's thesis. Note that mathematical texts tend to have fewer citations than other STEM fields, so this may have biased your expectations when looking through the literature.