PERF: Fix regression in datetime ops
HEAD:

```
In [1]: import pandas as pd; import numpy as np
In [2]: s = pd.Series(pd.to_datetime(np.arange(100000), unit='ms'))
In [3]: %timeit s - s.shift()
2.73 ms ± 30.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

0.21.0rc1:

```
527 ms ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

0.20.3

```
2.4 ms ± 57.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

xref https://github.com/pandas-dev/pandas/issues/17861