Performance regression in timeseries.DatetimeIndex.time_get
```python
import pandas as pd
import numpy as np

N = 100000
index = pd.date_range(start="2000", periods=N, freq="s", tz="US/Eastern")
%timeit index[0]
```

```
# pandas 1.0.2
12 µs ± 292 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# master
25.9 µs ± 935 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```

https://pandas.pydata.org/speed/pandas/index.html#timeseries.DatetimeIndex.time_get?p-index_type=%27tz_aware%27&commits=428791c5e01453ff6979b43d37c39c7315c0aaa2-1e5ff233c559ecb2f14fdb9c8d9537db3a02ce1d points to somewhere in `428791c5..1e5ff233`

```
* 1e5ff233c5 - ENH: allow passing freq=None to DatetimeIndex/TimedeltaIndex (#33635) (6 days ago) <jbrockmendel>
* 0907d9ece2 - BUG: DTI/TDI/PI.where accepting incorrectly-typed NaTs (#33715) (6 days ago) <jbrockmendel>
* ee1736a325 - REF: use cached inferred_type when calling lib.infer_dtype(index) (#33537) (6 days ago) <jbrockmendel>
* 23e91a7f3f - TST: groupby-reindex on DTI (#33638) (6 days ago) <CloseChoice>
* cb71376385 - BUG: support skew function for custom BaseIndexer rolling windows (#33745) (6 days ago) <Alex Kirko>
* e008a0a7ac - DOC: Remove ambiguity in fill_value documentation (#33788) (6 days ago) <Bharat Raghunathan>
* e82c0c69fd - CLN: Pass numpy args as kwargs (#33789) (6 days ago) <Daniel Saxton>
* a7741e3ff8 - PERF: op(frame, series) when series is not EA (#33600) (6 days ago) <jbrockmendel>
* ad4465edbd - CLN: remove unused PeriodEngine methods (#33796) (6 days ago) <jbrockmendel>
* 3c09d22f7f - BUG: arg validation in DTA/TDA/PA.take, DTI/TDI/PI.where (#33685) (6 days ago) <jbrockmendel>
* c6a1638bcd - BUG: Series[listlike_of_ints] incorrect on MultiIndex (#33539) (6 days ago) <jbrockmendel>
* 17dc6b0959 - REF: implement test_astype (#33734) (6 days ago) <jbrockmendel>
* c4ebf21e7e - REF: Implement NDArrayBackedExtensionArray (#33660) (6 days ago) <jbrockmendel>
* c60882429e - REF: remove need to override get_indexer_non_unique in DatetimeIndexOpsMixin (#33792) (6 days ago) <jbrockmendel>
* 0163b7941f - CLN: avoid getattr(obj, "values", obj) (#33776) (6 days ago) <jbrockmendel>
* 8c1df8d1b4 - BUG: Setting DTI/TDI freq affecting other indexes viewing the same data (#33552) (6 days ago) <jbrockmendel>
* 77a0f19c53 - ENH: Implement IntegerArray.sum (#33538) (6 days ago) <Daniel Saxton>
* f49269f5bf - ENH: Implement StringArray.min / max (#33351) (6 days ago) <Daniel Saxton>
* 31875eb3d8 - BUG: MonthOffset.name (#33757) (7 days ago) <jbrockmendel>
* 92afd5c2c0 - BUG: freq not retained on apply_index (#33779) (7 days ago) <jbrockmendel>
* ea09d504c9 - TST: prepare for freq-checking in tests.io (#33711) (7 days ago) <jbrockmendel>
* 2b6b89280b - TST: pd.NA TypeError in drop_duplicates with object dtype (#33751) (7 days ago) <Simon Hawkins>
* a585b63394 - TST: more accurate freq attr in `expected`s (#33773) (7 days ago) <jbrockmendel>
* f1fe03134d - CLN: simplify DTA.__getitem__ (#33714) (7 days ago) <jbrockmendel>
* 68132f3e0a - REGR: disallow mean of period column again (#33758) (7 days ago) <Joris Van den Bossche>
* 3c8593d0f3 - REF: simplify try_cast (#33764) (7 days ago) <jbrockmendel>
* 70175beb5b - REF: misplaced sort_index tests (#33774) (7 days ago) <jbrockmendel>
* 3df51fcc72 - DOC: data_columns can take bool (#33768) (7 days ago) <Kilian Lieret>
* 930a7f8373 - TYP/CLN: rogue type comment not caught by code checks (#33763) (7 days ago) <Simon Hawkins>
* 1a82659ec6 - TYP: construction (#33725) (7 days ago) <jbrockmendel>
```

(cc @jbrockmendel)