TST: Make tests for groupby median/mean more strict on dtype
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Followup to #41139.