How physically-based is the diffuse and specular distinction? The classical way of shading surfaces in real-time computer graphics is a combination of a (Lambertian) diffuse term and a specular term, most likely Phong or Blinn-Phong.



Now with the trend going towards physically-based rendering and thus material models in engines such as Frostbite, Unreal Engine or Unity 3D these BRDFs have changed. For example (a pretty universal one at that), the latest Unreal Engine still uses Lambertian diffuse, but in combination with the Cook-Torrance microfacet model for specular reflection (specifically using GGX/Trowbridge-Reitz and a modified Slick approximation for the Fresnel term). Furthermore, a 'Metalness' value is being used to distinguish between conductor and dielectric.

For dielectrics, diffuse is colored using the albedo of the material, while specular is always colorless. For metals, diffuse is not used and the specular term is multiplied with the albedo of the material.

Regarding real-world physical materials, does the strict separation between diffuse and specular exist and if so, where does it come from? Why is one colored while the other is not? Why do conductors behave differently?
To start, I highly suggest reading Naty Hoffman's Siggraph presentation covering the physics of rendering. That said, I will try to answer your specific questions, borrowing images from his presentation.

Looking at a single light particle hitting a point on the surface of a material, it can do 2 things: reflect, or refract. Reflected light will bounce away from the surface, similar to a mirror. Refracted light bounces around inside the material, and may exit the material some distance away from where it entered. Finally, every time the light interacts with the molecules of the material, it loses some energy. If it loses enough of its energy, we consider it to be fully absorbed.

To quote Naty, "Light is composed of electromagnetic waves. So the optical properties of a substance are closely linked to its electric properties." This is why we group materials as metals or non-metals.

Non metals will exhibit both reflection and refraction.


Metallic materials only have reflection. All refracted light is absorbed.


It would be prohibitively expensive to try to model the light particle's interaction with the molecules of the material. We instead, make some assumptions and simplifications.



If the pixel size or shading area is large compared to the entry-exit distances, we can make the assumption that the distances are effectively zero.
For convenience, we split the light interactions into two different terms. We call the surface reflection term "specular" and the term resulting from refraction, absorption, scattering, and re-refraction we call "diffuse".


However, this is a pretty large assumption. For most opaque materials, this assumption is ok and doesn't differ too much from real-life. However, for materials with any kind of transparency, the assumption fails. For example, milk, skin, soap, etc. 

A material's observed color is the light that is not absorbed. This is a combination of both the reflected light, as well as any refracted light that exits the material. For example, a pure green material will absorb all light that is not green, so the only light to reach our eyes is the green light.

Therefore an artist models the color of a material by giving us the attenuation function for the material, ie how the light will be absorbed by the material. In our simplified diffuse/specular model, this can be represented by two colors, the diffuse color, and the specular color. Back before physically-based materials were used, the artist would arbitrarily choose each of these colors. However, it should seem obvious that these two colors should be related. This is where the albedo color comes in. For example, in UE4, they calculate diffuse and specular color as follows:

DiffuseColor = AlbedoColor - AlbedoColor * Metallic;
SpecColor = lerp(0.08 * Specular.xxx, AlbedoColor, Metallic)


where Metallic is 0 for non-metals and 1 for metals. The 'Specular' parameter controls the specularity of an object (but it's usually a constant 0.5 for 99% of materials)