Why did we lose our fourth type of cone cell (in the eye)?
Let's be clear most mammals, including basal primates, had only 2 cone cell types. Primates evolved a new cone cell becasue frugivores have a distinct advantage being able to distinguish reds and yellows. The mammal lineage may have never had a fourth cone cell, if they did it was lost early.

Eutherian mammals lost the third cones cells likely because we spend several hundred million years as small nocturnal animals during the dominance of dinosaurs. in small nocturnal animals, there is a strong competition between having more rods for better low light vision vs having cones for color vision, so a loss of some cone cells may have been neutral or even beneficial. There was some work on bats that seemed to contradict this, but they failed to control for frugivory in the bat lineages they studied; So there is an issue with their results. There is a lot of variation in color vision across vertebrates so it is no surprise a consistent selective pressure could have a lasting effect. 

source