CLN: remove ensure_int_or_float
- [x] closes #27506
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

No need for a try/except; we can reason out which branch we go down in each of the places this is used.