CI: Unify pytest versions across builds
May be I'm missing something, but is there a reason to use different pytest versions in our CI builds? I see we're using those in the dependencies files:
- pytest
- pytest>=4.0.2
- pytest>=4.0.2,<5.0
- pytest=4.5.0
- pytest>=5.0.0
- pytest>=5.0.1

Since pytest is not a pandas dependency, but just a tool we use, feels like we should just use always the same version (the latest). All those (all versions since pytest 4) support Python >3.6, so I don't think there is any limitation that prevents this.

Am I missing something? Can we always use the same version?

I'm thinking on having a separate file for the tools needed for all builds (pytest, pytest-xdist...), and install both dependency files, the one of the build, and the common one. So we don't have to duplicate those in every file (it's faster to maintain and make changes, and also easier to see what dependencies we have and what we're testing in every build).

@jreback @TomAugspurger does this make sense?