MAINT: Correct small cast issues on Windows
Use correct types to avoid downcasts on Windows

- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
