Avoid importing from pandas at _libs files
- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Something that brought at to me at first [here](https://github.com/pandas-dev/pandas/pull/30395#discussion_r363101229)

---

cc @jbrockmendel 