ENH: Support downcasting of nullable dtypes in to_numeric
This currently does not yet work:

```
In [5]: pd.to_numeric(pd.Series([1, 2, 3], dtype="Int64"), downcast='integer') 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-64a04efaaac0> in <module>
----> 1 pd.to_numeric(pd.Series([1, 2, 3], dtype="Int64"), downcast='integer')

~/scipy/pandas/pandas/core/tools/numeric.py in to_numeric(arg, errors, downcast)
    179             for dtype in typecodes:
    180                 if np.dtype(dtype).itemsize <= values.dtype.itemsize:
--> 181                     values = maybe_downcast_to_dtype(values, dtype)
    182 
    183                     # successful conversion

~/scipy/pandas/pandas/core/dtypes/cast.py in maybe_downcast_to_dtype(result, dtype)
    141         dtype = np.dtype(dtype)
    142 
--> 143     converted = maybe_downcast_numeric(result, dtype, do_round)
    144     if converted is not result:
    145         return converted

~/scipy/pandas/pandas/core/dtypes/cast.py in maybe_downcast_numeric(result, dtype, do_round)
    234                     return new_result
    235             else:
--> 236                 if np.allclose(new_result, result, rtol=0):
    237                     return new_result
    238 

<__array_function__ internals> in allclose(*args, **kwargs)

~/miniconda3/envs/dev/lib/python3.7/site-packages/numpy/core/numeric.py in allclose(a, b, rtol, atol, equal_nan)
   2169 
   2170     """
-> 2171     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2172     return bool(res)
   2173 

<__array_function__ internals> in isclose(*args, **kwargs)

~/miniconda3/envs/dev/lib/python3.7/site-packages/numpy/core/numeric.py in isclose(a, b, rtol, atol, equal_nan)
   2268 
   2269     xfin = isfinite(x)
-> 2270     yfin = isfinite(y)
   2271     if all(xfin) and all(yfin):
   2272         return within_tol(x, y, atol, rtol)

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```

I suppose because under the hood a conversion to object dtype array is happening somewhere.