Pandas import breaks when setting filterwarnings
With released pandas (tested with 0.23.4), you get the following:

```
>>> import warnings
>>> warnings.filterwarnings('error')
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/joris/miniconda3/lib/python3.7/site-packages/pandas/__init__.py", line 26, in <module>
    from pandas._libs import (hashtable as _hashtable,
  File "/home/joris/miniconda3/lib/python3.7/site-packages/pandas/_libs/__init__.py", line 4, in <module>
    from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
  File "pandas/_libs/tslibs/conversion.pxd", line 11, in init pandas._libs.tslib
  File "pandas/_libs/tslibs/conversion.pyx", line 1, in init pandas._libs.tslibs.conversion
  File "pandas/_libs/tslibs/nattype.pyx", line 14, in init pandas._libs.tslibs.nattype
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
```

(you also get a long output with `warnings.filterwarnings('always')`).

This is not happening any more on master, so the reason seems to be fixed. 
But I was wondering, do we want to guarantee that this does not happen? (to the extent possible of course, meaning that we try to test this)