API: to_datetime with ordinals and no unit
xref #15828

Currently, if ordinals are passed to `to_datetime` without a unit, a `ns` resolution is assumed.  Given relative rarity of ns ordinals in the wild, I've been tripped up by this, e.g., below.  I wonder if it would be better to raise in the case of no unit, and force ordinal parsing to always be explicit?

```python
ordinals = pd.Series([1483228800000, 1483315200000, 1483401600000])

pd.to_datetime(ordinals)  # wrong
Out[49]: 
0   1970-01-01 00:24:43.228800
1   1970-01-01 00:24:43.315200
2   1970-01-01 00:24:43.401600
dtype: datetime64[ns]

pd.to_datetime(ordinals, unit='ms')
Out[51]: 
0   2017-01-01
1   2017-01-02
2   2017-01-03
dtype: datetime64[ns]
```