Correctly re-instate Matplotlib converters
Fixes https://github.com/pandas-dev/pandas/issues/27479. Converters should only be cached to be re-instated if they are original, and not pandas converters.

I'm struggling to write a test for this; I think it requires a clean state with no pandas imports, so one can check what was in the units registry before pandas was imported. I think this means a new test file with no pandas imports at the top, but maybe someone has a better idea?