BUG: to_datetime with xarray DataArray and specifie unit errors
- [x] closes #44053
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] coerce to numpy array before calling ```tslib.array_with_unit_to_datetime```function

Let me know if any change need to be done. Let me know if you need to make any changes.
I love the package hope I can start helping more :)