BLD: optional numpy import
The added workflow pip installs pandas without the requirements pre-installed. Seems to pull in requirements OK. (I also wanted to test with other versions of setuptools locally though before approving this PR)

OT for the sdist build step we do `python setup.py clean --quiet`. This fails if numpy not installed.

_Originally posted by @simonjayhawkins in https://github.com/pandas-dev/pandas/pull/38852#r552479485_