PERF: performance regression in replace() corner cases
ASV shows a *gigantic* regression (14629.25x) in a certain `replace` benchmark: https://pandas.pydata.org/speed/pandas/#replace.ReplaceList.time_replace_list?python=3.8&Cython=0.29.21&p-inplace=True&commits=07559156-dbee8fae

The simplified case is:

```
In [5]: df = pd.DataFrame({"A": 0, "B": 0}, index=range(4 * 10 ** 7))

In [6]: %timeit df.replace([np.inf, -np.inf], np.nan)
100 ms ± 6.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   # 1.1
1.2 s ± 31.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # master
```

Compared to 1.1, I don't see such a huge difference, but still a decent slowdown (x10).

Now, in this case, we have integer columns, but trying to replace infinity, which of course can never be present. So maybe before we had some shortcut for that. 
This also seems quite a cornercase, though. So not sure how critical the regression is.