PERF: Slowdown in IntervalIndex.get_loc
https://github.com/pandas-dev/pandas/pull/20611#issuecomment-437098669

```
asv find 7e6e7^..HEAD indexing.IntervalIndexing.time_loc_list
[...]
·· Installing 365eac4d <master~642> into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
··· Setting up indexing.py:220                                                                                                                                                                         ok
··· Running (indexing.IntervalIndexing.time_loc_list--).
··· indexing.IntervalIndexing.time_loc_list                                                                                                                                                      2.27±0ms
· Testing -----------------------------------O>-----------------------
·· For pandas commit f3160bfa <master~639>:
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
·· Building f3160bfa <master~639> for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt............................................................................................................
·· Installing f3160bfa <master~639> into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
··· Setting up indexing.py:220                                                                                                                                                                         ok
··· Running (indexing.IntervalIndexing.time_loc_list--).
··· indexing.IntervalIndexing.time_loc_list                                                                                                                                                    2.90±0.03s
· Testing -----------------------------------O------------------------
·· For pandas commit 610a19a3 <master~641>:
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
·· Building 610a19a3 <master~641> for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.................................................................................................................
·· Installing 610a19a3 <master~641> into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
··· Setting up indexing.py:220                                                                                                                                                                         ok
··· Running (indexing.IntervalIndexing.time_loc_list--).
··· indexing.IntervalIndexing.time_loc_list                                                                                                                                                    2.92±0.01s.
· Greatest regression found: 610a19a3 <master~641>
```

cc @qwhelan 

We seem to spend a lot of time in Series._set_axis and `is_all_dates` for some reason.