BUG: panda3d/core.cpython-39-darwin.so is not code signed on macOS Big Sur
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [x] (optional) I have confirmed this bug exists on the master branch of pandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

On macOS Big Sur 11.2.3 on MacBook Pro M1, with Python 3.9.2, pandas _master_ or _v1.2.3_ branch:

```bash
% python3.9 -c "import panda3d.core"                                              
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/panda3d/core.cpython-39-darwin.so, 2): no suitable image found.  Did find:
	/opt/homebrew/lib/python3.9/site-packages/panda3d/core.cpython-39-darwin.so: code signature in (/opt/homebrew/lib/python3.9/site-packages/panda3d/core.cpython-39-darwin.so) not valid for use in process using Library Validation: Trying to load an unsigned library
```

Note below that pandas itself was compiled with a signature.

#### Problem description

`panda3d/core.cpython-39-darwin.so` is not code signed on macOS Big Sur

```bash
% codesign -v -v /opt/homebrew/lib/python3.9/site-packages/panda3d/core.cpython-39-darwin.so
/opt/homebrew/lib/python3.9/site-packages/panda3d/core.cpython-39-darwin.so: code object is not signed at all
In architecture: arm64
```

#### Expected Output

`import panda3d.core` should load as well.

#### Output of ``pd.show_versions()``

```bash
% python3.9 -c "import pandas; pandas.show_versions()"
```

<details>

INSTALLED VERSIONS
------------------
commit           : 921477ed71b93b9f0162fd018431f1076d48b37f
python           : 3.9.2.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 20.3.0
Version          : Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : nl_NL.UTF-8
LOCALE           : nl_NL.UTF-8

pandas           : 1.3.0.dev0+999.g921477ed71
numpy            : 1.20.1
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.0.1
setuptools       : 53.0.0
Cython           : 0.29.22
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.3
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.4
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.6.0
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

</details>
