How are colors really mixed? I'm kind of stuck on this one. When following a conventional, high-level shading tutorial, you'll come across these images that are like ambient + diffuse + specular = result, but I can tell that it's not straight addition (otherwise the picture would seem overexposed).



What's the "formula" for mixing two rgb colors?
With an old-style Phong shader (as in your example), they really are just added together. Each of those things is a contribution to the overall amount of light, so they're just added together, the same way you add together the contribution from each light source.

Your intuition is right that you often end up with overexposed specular highlights with this style of shading. In a modern, physically-based pipeline, it's common to instead use an energy-conserving shader. The contributions of each layer (specular, clear-coat/lacquer, diffuse, subsurface) are still added together, but the amount of light in each layer reduces the input amount of light for the next layer. That is, if the surface is giving a strong specular reflection, it's reflecting most of the light, so less light is available for the diffuse layer. In the same way, any light that's reflected by the diffuse layer reduces the amount of light available for subsurface scattering.