BUG: pd.to_datetime cannot infer and coerce dates with AM/PM and infer_datetime_format=True and errors="coerce".
#### Checklist
- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandas.
- [x] (optional) I have confirmed this bug exists on the master branch of pandas.

---

#### Code Sample

```python
series = pd.Series(data=['1/1/2021 11:00:00 AM', '1/1/2021 12:00:00 PM', '1/1/2021 1:00:00 PM'])
result = pd.to_datetime(series, errors='coerce', infer_datetime_format=True)
```

#### Problem description
The function should be able to infer datetime strings with AM/PM correctly.  This can be seen when using errors='raise' which will correctly return the expected values.  Looks like this is a result of an existing issue - https://github.com/pandas-dev/pandas/issues/25143.

#### Expected Output
expected = pd.Series(data=[Timestamp('2021-01-01 11:00:00'), Timestamp('2021-01-01 12:00:00'), Timestamp('2021-01-01 13:00:00')])

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : 6a683a2911bd82ea40c7e11b8135a0e952f55f2c
python           : 3.8.10.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.6.0
Version          : Darwin Kernel Version 19.6.0: Tue Jun 22 19:49:55 PDT 2021; root:xnu-6153.141.35~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : en_US.UTF-8
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8
pandas           : 1.4.0.dev0+597.g6a683a2911
numpy            : 1.21.1
pytz             : 2021.1
dateutil         : 2.8.2
pip              : 21.1.3
setuptools       : 52.0.0.post20210125
Cython           : 0.29.24
pytest           : 6.2.4
hypothesis       : 6.14.4
sphinx           : 4.1.2
blosc            : 1.10.4
feather          : None
xlsxwriter       : 1.4.4
lxml.etree       : 4.6.3
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 3.0.1
IPython          : 7.25.0
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : 1.3.2
fsspec           : 2021.05.0
fastparquet      : 0.6.3
gcsfs            : 2021.05.0
matplotlib       : 3.4.2
numexpr          : 2.7.3
odfpy            : None
openpyxl         : 3.0.7
pandas_gbq       : None
pyarrow          : 4.0.1
pyxlsb           : None
s3fs             : 2021.05.0
scipy            : 1.7.0
sqlalchemy       : 1.4.22
tables           : 3.6.1
tabulate         : 0.8.9
xarray           : 0.18.2
xlrd             : 2.0.1
xlwt             : 1.3.0
numba            : 0.53.1

</details>
