Use conda-env for CI environments
Closes https://github.com/pandas-dev/pandas/issues/20720

The old setup where things were installed in multiple places resulted in some strange environments, with multiple copies of the same package in `site-packages`.

This attempts to reduce the environment definition to a single file per CI job, and the environment installation to 3 steps

- create the environment
- uninstall pandas if it's present (installing a pandas dependency)
- build pandas from source & install

Travis only for now. Will move over the appveyor / circle ones too.