PERF: Fix performance regression in infer_dtype
Fixes major performance regression in `infer_dtype` introduced in
aaaac86ee019675119cb0ae9c3fb7a2b7eef9959

- [x] closes #28814 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
