iOS Smart Invert dulls colors
This may or may not be related to how smart invert and neither night shift NOR true tone can operate together. Try your comparison with night shift off?

Edit: I tested it some more, and although true tone being off affects the tone a lot, the visual hues of non-inverted items in smart invert do look a lot more desaturated and just generally poopy. I think there is some incorrect colorspace handling going on in here. 

When I take a screenshot in smart invert mode, the actual saved image appears to be truly inverted. this seems to indicate to me that under the hood everything does get inverted, and smart invert actually goes in and re-inverts some of the things back that it thinks shouldn't have gotten inverted. This process is likely what fails to bring colors back to their original vibrance.