 BUG: don't mangle NaN-float-values and pd.NaT (GH 22295) 
it is more or less the clean-up after PR #21904 and PR #22207, the underlying hash-map handles all cases correctly out-of-the box and thus no special handling is needed.

As a collateral, the mangling of NaNs and NaT is fixed.

- [x] closes #22295 
- [x] closes #22332
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
