What is the most physically accurate representation of color possible in computer graphics?
The most physically accurate way would be to have a $l(\theta)$ which for each possible color frequency has a certain value.

Converting to RGB would then need a frequency responce function for each channel and the result is then $\int_{infrared}^{ultraviolet} l(\theta)F_{red}(\theta)d\theta$.

Reflected light then has a 2 dimensional response function: $L_{reflected}(\theta) = \int l_{incoming}(\phi)R(\phi,\theta)d\phi$. (ignoring the BRDF here for simplicity)

Drawbacks are that instead of just 3 color channels you now have infinite channels to worry about.