CI: isort ci
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

asv_bench and doc dir where [added](https://github.com/pandas-dev/pandas/pull/24092/files) to the isort skip section. We should actually check this directory in CI.

Checking doc dir is more effort that its worth ( barely any .py files ) so let's just leave this.