BUG: behavior change in dataframe.shift(, freq=infer) when upgrading from pandas 1.1.5 to 1.2.x
- [x]  I have checked that this issue has not already been reported.

- [x]  I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

#### Code Sample, a copy-pastable example

```python
import pandas as pd
import numpy as np
import io

# replicate the typical data I am reading with pd.rea_csv
dates=np.arange(0,600.1,0.1)
dateString="time, count\n"
for  count, time in enumerate(dates):
    dateString=dateString+"{:0.3f}, {}\n".format(time, count)
# create dataframe and convert 'time' as a datetime and put it to index.
df=pd.read_csv(io.BytesIO(dateString.encode()),sep=',')
df['date']=pd.to_datetime(df.time, unit='s')
df.set_index('date', inplace=True, drop=False)
df.shift(1, freq='infer') # <-- this is working with pandas 1.1.5 but not any more with 1.2.0 and above
```
```python-traceback
Traceback (most recent call last):
  File "debugPandas.py", line 23, in <module>
    df.shift(1, freq='infer')
  File "\lib\site-packages\pandas\core\frame.py", line 4600, in shift
    return super().shift(
  File "\lib\site-packages\pandas\core\generic.py", line 9453, in shift
    raise ValueError(msg)
ValueError: Freq was not set in the index hence cannot be inferred
```

#### Problem description

I am reading time series results from an other software with `pd.read_csv` and I put the time vector converted as datetime in index. At some stage in the data processing I need to use `dataframe.shift` using `freq='infer'`. This was working fine with pandas 1.1.5 but stopped working when upgrading to pandas 1.2.0 getting a `ValueError`. I could not find any reference of this in the change log so I guess it might be considered as a regression.
I worked it around forcing the proper frequency in the shift function instead of using `infer`

#### Investigations

<details>

Trying to go into more depth in the issue using the above dataframe `df`
```python
print(df.index.inferred_freq)
print(pd._libs.algos.unique_deltas(df.index.asi8))
```
returns for pandas 1.1.5
```
100L
[100000000]
```
and for pandas 1.2.x
```
None
[ 99999999 100000000 100000001]
```

There is some kind of rounding issue when converting the floats serie to datatime serie. This is not the case if the index is created directly from the `dates` ndarray so there might be something with the `read_csv` part.
This is working fine waterver version of pandas:

```python
dates=np.arange(0,600.1,0.1)
df=pd.DataFrame(data=dates, columns=['time'])
df['date']=pd.to_datetime(df.time, unit='s')
df.set_index('date', inplace=True, drop=False)
df.shift(1, freq='infer')  # <-- works with pandas 1.2.x
print(df.index.inferred_freq) # --> 100L
print(pd._libs.algos.unique_deltas(df.index.asi8)) # --> [100000000]
```


</details>

#### Output of ``pd.show_versions()``

for pandas 1.2.3

<details>

INSTALLED VERSIONS
------------------
commit           : f2c8480af2f25efdbd803218b9d87980f416563e
python           : 3.9.1.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.14393
machine          : AMD64
processor        : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : English_United States.1252

pandas           : 1.2.3
numpy            : 1.20.1
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.0.1
setuptools       : 52.0.0.post20210125
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : 7.21.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.4
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

</details>

for pandas 1.1.5
<details>

INSTALLED VERSIONS
------------------
commit           : b5958ee1999e9aead1938c0bba2b674378807b3d
python           : 3.9.1.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.14393
machine          : AMD64
processor        : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : English_United States.1252

pandas           : 1.1.5
numpy            : 1.20.1
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.0.1
setuptools       : 52.0.0.post20210125
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : 7.21.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.4
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

</details>