DEPS: Pinning versions of some dependencies to speed up environment creation
- [X] closes #24012
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Conda is taking 45 minutes to create an environment with our `environment.yml`. While this is a bug in conda, it won't be fixed until the next version. So, a workaround for now is to pin the verion of some of the packages.

I'm also removing couple of packages that I don't think are useful in a local environment, or in the CI build that will use the file (for the linting and to build the docs).