Performance regression in TimedeltaIndexing.time_get_loc
Setup:

```python
index = pd.timedelta_range(start="1985", periods=1000, freq="D")
timedelta = index[500]

%timeit index.get_loc(timedelta)
```

```
# master
7.39 µs ± 78.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# 1.0.3
2.82 µs ± 13.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```

See https://pandas.pydata.org/speed/pandas/#timedelta.TimedeltaIndexing.time_get_loc?commits=9929fca8-62c7dd3e

Happened somewhere between May 11 and May 27 (the gap in the benchmarks)