Taxonomy term not displayed in the current language
The language of the parent entity is passed to entity formatters and they respect that when displaying referenced entities.

If you want to override that, you can for example provide your own entity formatter plugin that overriddes \Drupal\Core\Field\Plugin\Field\FieldFormatter\EntityReferenceFormatterBase::getEntitiesToView() and ignores the passed in $langcode. There is no configuration for that.

Since you probably want the entity label to be displayed, you should subclass \Drupal\Core\Field\Plugin\Field\FieldFormatter\EntityReferenceLabelFormatter.

Alternatively, you can also do something in preprocess/twig.