BUG: ValueError: Cannot convert non-finite values (NA or inf) to integer only when DF exceed certain size
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [x] (optional) I have confirmed this bug exists on the master branch of pandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### 

Here is the code in question:

```
    print(df.shape)
    print(df.memory_usage().sum())
    df.dropna(inplace=True)  # drop rows with not enough lookback (they will be nan)
```
The line `dropna` line is what throws the error

So when i run my code with adding 500 features/columns the output is following

```
(2177432, 503)
4398412768
```

However when i run the same exact code for 1000 features

```
(2177432, 1003)
8753276768
```

I get an error

```
  File "C:\Users\Ben\PycharmProjects\tradingbot\tradingbot\trainer\sample_maker.py", line 17, in get_signal_features
    df.dropna(inplace=True)  # drop rows with not enough lookback (they will be nan)
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\frame.py", line 4751, in dropna
    count = agg_obj.count(axis=agg_axis)
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\frame.py", line 7807, in count
    return result.astype("int64")
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\generic.py", line 5698, in astype
    new_data = self._data.astype(dtype=dtype, copy=copy, errors=errors)
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\internals\managers.py", line 582, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\internals\managers.py", line 442, in apply
    applied = getattr(b, f)(**kwargs)
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\internals\blocks.py", line 625, in astype
    values = astype_nansafe(vals1d, dtype, copy=True)
  File "C:\Users\Ben\PycharmProjects\tradingbot\venv\lib\site-packages\pandas\core\dtypes\cast.py", line 868, in astype_nansafe
    raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
ValueError: Cannot convert non-finite values (NA or inf) to integer
```

The version i am running is 1.0.5