TST: Reduce number of numba tests run
- xref https://github.com/pandas-dev/pandas/pull/41971#issuecomment-860607472
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

