BUG: ValueError: Cannot convert non-finite values (NA or inf) to integer only when DF exceed certain size
- [ ] closes #35227
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
