Negated str.isnumeric() fails on column with None when called together with notnull() in the same boolean expression
#### Code Sample

Create a dataframe:
```
import pandas as pd
df = pd.DataFrame({'A': ['0', '1', '2', 'X', None]})
```

Obviously this fails:
```
print(df[df.A.str.isnumeric()])
```

So I can do this and it works:
```
print(df[df.A.notnull() & df.A.str.isnumeric()])
```

However when negating the `str.isnumeric()` with unary `~` it fails:
```
print(df[df.A.notnull() & ~df.A.str.isnumeric()])
```
Error message:
>TypeError: bad operand type for unary ~: 'NoneType'

Full error:

<details>

> print(df[df.A.notnull() & ~df.A.str.isnumeric()])
> Traceback (most recent call last):
> 
>   File "<ipython-input-32-9d35109941d7>", line 1, in <module>
>     df[df.A.notnull() & ~df.A.str.isnumeric()]
> 
>   File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\core\generic.py", line 1473, in __invert__
>     new_data = self._data.apply(operator.invert)
> 
>   File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\core\internals\managers.py", line 440, in apply
>     applied = b.apply(f, **kwargs)
> 
>   File "C:\Users\My.Name\AppData\Local\Continuum\miniconda3\envs\main\lib\site-packages\pandas\core\internals\blocks.py", line 390, in apply
>     result = func(self.values, **kwargs)
> 
> TypeError: bad operand type for unary ~: 'NoneType'

</details>

When doing it in two step it works:
```
dft = df[df.A.notnull()].copy()
print(dft[~dft.A.str.isnumeric()])
```

#### Problem description

It's inconsistent for the combined expression to fail because of the unary `~`.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : en
LOCALE           : None.None

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 46.0.0.post20200308
Cython           : 0.29.15
pytest           : None
hypothesis       : None
sphinx           : 2.4.4
blosc            : None
feather          : None
xlsxwriter       : 1.2.8
lxml.etree       : 4.5.0
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.13.0
pandas_datareader: None
bs4              : 4.8.2
bottleneck       : None
fastparquet      : 0.3.3
gcsfs            : None
lxml.etree       : 4.5.0
matplotlib       : 3.2.0
numexpr          : None
odfpy            : None
openpyxl         : 3.0.3
pandas_gbq       : None
pyarrow          : 0.16.0
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : 1.3.13
tables           : None
tabulate         : 0.8.6
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : 1.2.8
numba            : 0.48.0

</details>
