PERF: improve Integer/BooleanArray astype to numeric dtypes
Currently, in the IntegerArray or BooleanArray `astype`, we first convert to object dtype, and then convert to the requested dtype. 
For example, in `IntegerArray.astype`:

https://github.com/pandas-dev/pandas/blob/7d7f885856b0c3d51eaf15beaac9d4f30c23797d/pandas/core/arrays/integer.py#L551-L553

where `self._coerce_to_ndarray()` creates an object ndarray. 
In case of converting to integer (and if there are no NaNs), or in case of converting to float, this roundtrip through object dtype is not needed (we can directly convert the `_data` and in case of float set NaNs)
