Improve performance of pandas arrow extractor
Looks like this change
```
pa_table[pa_table.column_names[0]].to_pandas(types_mapper=pandas_types_mapper)
```
doesn't return a Series with the correct type.
This is related to https://issues.apache.org/jira/browse/ARROW-9664

Since the types_mapper isn't taken into account, the ArrayXD types are not converted to the correct pandas extension dtype @lhoestq I think I found a workaround... 😉  For some reason the benchmarks are not run Oo Anyway, merging.
We'll see on master how much speed ups we got
While reviewing PR #2505, I noticed that pandas arrow extractor could be refactored to be faster.