deregister_matplotlib_converters() in 0.25 incorrectly removes Matplotlib converters 
Consider

```python
import matplotlib.units as munits
import pandas as pd
pd.plotting.deregister_matplotlib_converters()

print(pd.__version__)
for key in munits.registry:
    print(key, '\t', munits.registry[key])
```

`deregister_matplotlib_converters()` should only remove `pandas` converters, and not any other converters in the MPL units registry, but in pandas 0.25 it removes some of the Matplotlib converters.

#### Problem description

For pandas 0.24 this prints:
```
0.24.2
<class 'decimal.Decimal'> 	 <matplotlib.units.DecimalConverter object at 0x1135515f8>
<class 'datetime.datetime'> 	 <matplotlib.dates.DateConverter object at 0x11ed79cc0>
<class 'datetime.date'> 	 <matplotlib.dates.DateConverter object at 0x11ed79c88>
<class 'numpy.datetime64'> 	 <matplotlib.dates.DateConverter object at 0x11ed79c50>
```
But for pandas 0.25 it gives
```
0.25.0
<class 'decimal.Decimal'> 	 <matplotlib.units.DecimalConverter object at 0x1133cd588>
```

This means `pandas.plotting.deregister_matplotlib_converters` is removing some of the built in matplotlib converters from the units registry when it shouldn't be; this is causing some of our tests to break upstream (xref https://github.com/matplotlib/matplotlib/issues/14859)
