BUG: DataFrame improperly storing ints as floats
- [ X ] I have checked that this issue has not already been reported.

- [ X ] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---
#### Code Sample, a copy-pastable example

```python
import pandas as pd

seed = [{'start': 0, 'end': 0, 'dist': 5900.22782978315}, {'start': 0, 'end': -1, 'dist': 0.0}, {'start': -1, 'end': 0, 'dist': 14258.70864551041}, {'start': -1, 'end': -1, 'dist': 8397.882580246285}]
df = pd.DataFrame(seed)
print('Original dtypes:')
print(df.dtypes)
df['start']=df['start'].astype(int)
df['end']=df['end'].astype(int)
print('\r\nNew column dtypes:')
print(df.dtypes)
print('\r\nValue dtypes:')
row = df.iloc[0]
print(f"A: {row['start'].dtype}")
print(f"B: {row['end'].dtype}")

```

#### Observed output
Original dtypes:
start      int64
end        int64
dist     float64
dtype: object  

New column dtypes:
start      int64
end        int64
dist     float64
dtype: object  

Value dtypes:
A: float64
B: float64

#### Problem description

It seems that under some circumstances (I'm not sure what's unique about this particular case), pandas is storing integer columns as floats even when there is not a missing/NaN value. The column type is listed as int64, doing an `astype(int)` yields an int64 column, but the actual value dtypes are float64. They should be int64.

#### Expected Output
Original dtypes:
start      int64
end        int64
dist     float64
dtype: object  

New column dtypes:
start      int64
end        int64
dist     float64
dtype: object  

Value dtypes:
A: **int64**
B: **int64**

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.7.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.5.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.5
numpy            : 1.18.5
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.1.1
setuptools       : 47.3.1.post20200622
Cython           : 0.29.20
pytest           : 5.4.3
hypothesis       : 5.18.3
sphinx           : 3.1.1
blosc            : None
feather          : None
xlsxwriter       : 1.2.9
lxml.etree       : 4.5.1
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.16.1
pandas_datareader: None
bs4              : 4.9.1
bottleneck       : 1.3.2
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.5.1
matplotlib       : 3.2.2
numexpr          : 2.7.1
odfpy            : None
openpyxl         : 3.0.3
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.4.3
pyxlsb           : None
s3fs             : None
scipy            : 1.5.0
sqlalchemy       : 1.3.18
tables           : 3.6.1
tabulate         : None
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.2.9
numba            : 0.50.1

</details>
