Revert unnecessary changes from 22019
At the time, https://github.com/pandas-dev/pandas/pull/22019 seemed to
break API in a few places. Since then, those API breaking changes have
disappeared, so the changes to the tests and the release note are no
longer necessary.

This restores the tests to the 0.23.4 version.

Closes https://github.com/pandas-dev/pandas/issues/24789