BUG: Avoid casting to double type unnecessarily when setting values i…
…n time delta column

- [x] closes #23511
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

When setting a timedelta column in a dataframe using `pd.Series` or array as index, there is some rounding error because the underlying TimeDeltaBlock is casted to double and back to int. This patch fixes the issue. 

