Entity Reference: Filter by an entity reference view not working as expected
You can't at the moment.

See \Drupal\views\Plugin\EntityReferenceSelection\ViewsSelection::getReferenceableEntities(), it just uses the view to get the results and then calls $entity->label(). Which means you have no way to override that.

Which means that if you want this, you need to provide your own plugin. You can either subclass from ViewsSelection and still a view, or just extend from the default and override that method.

Might also be worth opening a feature request, I've already had this use case multiple times, and a new checkbox or option somewhere would be a nice, backwards compatible way to introduce this feature in a minor release.