TimedeltaBlock.fillna treats int as second
Seems like assuming nanosecond would be more natural, or just refusing to assume.

https://github.com/pandas-dev/pandas/blob/master/pandas/core/internals.py#L1981
