PERF: unnecessary casting in merge
### 

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this issue exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas.

- [X] I have confirmed this issue exists on the master branch of pandas.


### Reproducible Example

In reshape.merge we do some some casting motivated by what dtypes HashTable supports, see the comment '# upcast 'by' parameter because HashTable is limited'.  That comment is no longer accurate, and we can avoid some of this casting.  Something like core.algorithms._ensure_data may be reusable.

### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>


### Prior Performance

_No response_