Revert "Backport PR #44204 on branch 1.3.x (CI: Python Dev build)"
Reverts pandas-dev/pandas#44208

THIS IS ONLY ON 1.3.x.

Let's try reverting this. In theory, the test changes that caused the timeouts shouldn't be backported, so this shouldn't be needed.