PERF: regression in IntervalIndex intersection
There were several regressions related to the set op methods of IntervalIndex. For example https://github.com/pandas-dev/pandas/pull/41929#issuecomment-860514675 which was fixed (and backported) in https://github.com/pandas-dev/pandas/pull/42197. 

But as far as I can see, that didn't fix the regression in https://pandas.pydata.org/speed/pandas/#index_object.IntervalIndexMethod.time_intersection?python=3.8&Cython=0.29.21&p-param1=100000&commits=de07087a-6f953a8c (50x slow down of `index_object.IntervalIndexMethod.time_intersection(100000)`) 
This one might be caused by one of the other interval index related commits in the range (#41832, #41824, #41883)

cc @jbrockmendel 
