Error in matplotlib with datetime64 with pandas 0.21.0
With `pandas` 0.21 this code with plotting `datetime64` fails:

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

fig, sub = plt.subplots()
x = np.array(['2015-07-13T04:32:23+0200', '2015-07-13T04:32:24+0200', '2015-07-13T04:32:25+0200'], dtype="datetime64")
y = np.arange(len(x))
sub.plot(x, y)
sub.set_xlim(xmin=x[0], xmax=x[-1])
```

```
TypeError                                 Traceback (most recent call last)
<ipython-input-4-483d578e8eb2> in <module>()
      7 y = np.arange(len(x))
      8 sub.plot(x, y)
----> 9 sub.set_xlim(xmin=x[0], xmax=x[-1])

~\Anaconda3\lib\site-packages\matplotlib\axes\_base.py in set_xlim(self, left, right, emit, auto, **kw)
   2917                  'in singular transformations; automatically expanding.\n'
   2918                  'left=%s, right=%s') % (left, right))
-> 2919         left, right = mtransforms.nonsingular(left, right, increasing=False)
   2920 
   2921         if self.get_xscale() == 'log' and (left <= 0.0 or right <= 0.0):

~\Anaconda3\lib\site-packages\matplotlib\transforms.py in nonsingular(vmin, vmax, expander, tiny, increasing)
   2892     close to zero, it returns -*expander*, *expander*.
   2893     '''
-> 2894     if (not np.isfinite(vmin)) or (not np.isfinite(vmax)):
   2895         return -expander, expander
   2896 

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```

While 0.20.3 works fine and produces the result.

I'm not sure if it is a `matplotlib` or `pandas`.

Issue in matplotlib: [#9771](https://github.com/matplotlib/matplotlib/issues/9771)