Inconsistent pd.to_datetime() behaviour dependent on dtype
I have two Series, both containing years. One series is `dtype == float64` and the other is `dtype == object`. When I apply pd.to_datetime() to these two columns, i get drastically different results.

```python
## Pandas 0.24.1
import pandas as pd
year_string = pd.Series(['1980','1985','1990','1995','2000'])
year_float = pd.Series([1980.0, 1985.0, 1990.0, 1995.0, 2000.0])

## Works as expected
pd.to_datetime(year_string, infer_datetime_format=True)
# 0   1980-01-01
# 1   1985-01-01
# 2   1990-01-01
# 3   1995-01-01
# 4   2000-01-01
# dtype: datetime64[ns]

## Automatically treats floats epoch nanoseconds
pd.to_datetime(year_float, infer_datetime_format=True)
# 0   1970-01-01 00:00:00.000001980
# 1   1970-01-01 00:00:00.000001985
# 2   1970-01-01 00:00:00.000001990
# 3   1970-01-01 00:00:00.000001995
# 4   1970-01-01 00:00:00.000002000
# dtype: datetime64[ns]
```

While I understand why this difference exists (perhaps there's an assumption being made that if dtype == float, then the value are epoch nanoseconds?), I'm not sure I agree with the inconsistency in behavior based strictly on dtype. I would favor a solution that treated the float64 series the same as the string/object series. Is it possible that the logic that infers `year_string` as containing years could also be applied to `year_float`?