Cleanup CI environment files after #27542
https://github.com/pandas-dev/pandas/pull/27542 got CI passing again by removing matplotlib from a couple of the slow travis jobs.

We should verify that

1. We still have adequate matplotlib coverage (I think we do)
2. We can remove the pytest and pytest-xdist pins

Also monitor https://travis-ci.community/t/builds-often-fail-because-of-lack-of-resources-memory-cpu/4331 to see if travis brings back higher memory workers.