ENH: Use IntergerArray to avoid forced conversion from integer to float
- [ ] closes #xxxx
- [x] tests added / passed
- [ ] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is a new attempt to solve issue #16918 after an attempt with PR #27073