REGR: re-evaluate merge fix of  PR #24916
See my comments and the possible regression on PR https://github.com/pandas-dev/pandas/pull/24916 (code review https://github.com/pandas-dev/pandas/pull/24916#pullrequestreview-196792659).

We can maybe have the code discussion there on the diff, but opening this issue as a reminder that we need to fix this for 0.25.0.