PERF: Regressions since 0.20.3
These are fairly reproducible on my machine:

```
       before           after         ratio
     [eac4d3f7]       [92db5c9c]
+           396μs            746μs     1.88  inference.DtypeInfer.time_uint32
+     2.63±0.02ms       3.93±0.1ms     1.49  indexing.IndexingMethods.time_take_dtindex
+     2.64±0.06ms      3.83±0.06ms     1.45  indexing.IndexingMethods.time_take_intindex
```

```
       before           after         ratio
     [eac4d3f7]       [92db5c9c]
+     1.43±0.01ms      1.74±0.07ms     1.22  frame_ctor.FrameConstructorDTIndexFromOffsets.time_frame_ctor('BDay', 1)
+     1.56±0.02ms      1.74±0.07ms     1.12  frame_ctor.FrameConstructorDTIndexFromOffsets.time_frame_ctor('MonthBegin', 2)
+          56.0ms           61.8ms     1.10  frame_ctor.FrameConstructorDTIndexFromOffsets.time_frame_ctor('FY5253Quarter_2', 2)
```
