Cannot infer datetime dtype
```python
import pandas as pd
from pandas._libs import lib

test = pd.date_range("20110101", freq="s", periods=10).values
lib.infer_dtype(test, skipna = False)

print('Trying to infer datetime dtype:')
lib.infer_dtype(test.astype('O'), skipna = False)
lib.infer_dtype(test.tolist(), skipna = False)

lib.infer_datetimelike_array(test.astype('O'))
lib.infer_datetimelike_array(test.tolist())

lib.maybe_convert_objects(test.astype('O')).dtype
```
```python
'datetime64'

Trying to infer datetime dtype:
'integer'
'integer'
'mixed'
'mixed'
dtype('int64')
```
I'm not sure that all of these should produce 'datetime' but any of should.