Int64 Series input to numpy.polyfit() fails 
#### Code Sample, a copy-pastable example if possible

```python
import numpy as np
import pandas as pd

x = pd.Series([1,2,3]).astype('Int64')
y = pd.Series([4,4,6])

np.polyfit(x,y,1)
```
#### Problem description

The code raises from within Numpy:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<__array_function__ internals>", line 6, in polyfit
  File "C:\Anaconda3\lib\site-packages\numpy\lib\polynomial.py", line 609, in polyfit
    rcond = len(x)*finfo(x.dtype).eps
  File "C:\Anaconda3\lib\site-packages\numpy\core\getlimits.py", line 381, in __new__
    raise ValueError("data type %r not inexact" % (dtype))
ValueError: data type <class 'numpy.object_'> not inexact
```
However, the same Series with ``dtype=int64`` do not raise. This happens for both ``pandas=0.24.0`` and ``pandas=1.0.3``

Since this dtype is documented as experimental I opened it as a pandas issue. Not sure if this is related to #29738 .

#### Expected Output
```
array([1.        , 2.66666667])
```
#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Windows
OS-release       : 7
machine          : AMD64
processor        : Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 1.0.3
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.2.0.post20200209
Cython           : None
pytest           : 4.5.0
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.13.0
pandas_datareader: None
bs4              : 4.8.2
bottleneck       : 1.3.0
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.2.0
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 4.5.0
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

xref [https://github.com/mwaskom/seaborn/issues/1971](https://github.com/mwaskom/seaborn/issues/1971)
cc @mojones who opened the linked seaborn issue