BUG: to_datetime very slow with unsigned ints for unix seconds
- [x] I have checked that this issue has not already been reported.

- [ ] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

I had a data pipeline that was terribly slow. Turns out the issue was that all the time was spent in pd.to_datetime calls with unix sec integers as input, because I was passing big-endian ints. With normal ints it's about 1000x faster.

Can anyone reproduce this performance issue?
is it possible to improve on this "gotcha", e.g. by forcing a typecast on input, or even some other way that doesn't require a copy and extra memory?

```
In [10]: %timeit index = pd.to_datetime(np.arange(1_000_000), unit="s", utc=True)
7.83 ms ± 22 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [23]: %time index = pd.to_datetime(np.arange(1_000_000).astype("uint32"), unit="s", utc=True)
CPU times: user 4.03 s, sys: 12.5 ms, total: 4.05 s
Wall time: 4.04 s

In [24]: %time index = pd.to_datetime(np.arange(1_000_000).astype("uint64"), unit="s", utc=True)
CPU times: user 8.42 s, sys: 12.9 ms, total: 8.44 s
Wall time: 8.45 s

In [25]: %time index = pd.to_datetime(np.arange(1_000_000).astype("int64"), unit="s", utc=True)
CPU times: user 13.1 ms, sys: 4.05 ms, total: 17.2 ms
Wall time: 15.9 ms

In [26]: %time index = pd.to_datetime(np.arange(1_000_000).astype("int32"), unit="s", utc=True)
CPU times: user 13.4 ms, sys: 4.54 ms, total: 18 ms
Wall time: 16.5 ms
```

#### Output of ``pd.show_versions()``

<details>

In [11]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : 2cb96529396d93b46abab7bbc73a208e708c642e
python           : 3.8.10.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 20.5.0
Version          : Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.UTF-8

pandas           : 1.2.4
numpy            : 1.20.3
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.1.2
setuptools       : 49.6.0.post20210108
Cython           : None
pytest           : 6.2.4
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.6 (dt dec pq3 ext lo64)
jinja2           : 3.0.1
IPython          : 7.24.1
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : 1.3.2
fsspec           : 2021.05.0
fastparquet      : None
gcsfs            : None
matplotlib       : 3.4.2
numexpr          : None
odfpy            : None
openpyxl         : 3.0.7
pandas_gbq       : None
pyarrow          : 4.0.1
pyxlsb           : None
s3fs             : None
scipy            : 1.6.3
sqlalchemy       : 1.4.18
tables           : None
tabulate         : 0.8.9
xarray           : 0.18.2
xlrd             : 1.2.0
xlwt             : None
numba            : 0.53.1

</details>
