TST: Make ARM build work (not in the CI)
- [X] closes #41737

This seems to be working, see:

https://app.circleci.com/pipelines/github/datapythonista/pandas/11/workflows/2ca0487f-5a63-4f01-936c-e83827db714f/jobs/11

But tests are failing. I opened #41740 the the failures.