FIX: revert part of #11770
Reverts part of:

https://github.com/pydata/pandas/pull/11770

Closes:

https://github.com/matplotlib/matplotlib/issues/6365

The mistake in #11770 was missing that pandas had a 1/us not 1/s
scaled bucket.
