Hide taxonomy term from a view filter where the filter results are empty
In Views, there is an option to specify a relationship and to require that relationship. You can find it in your View's config page, under Advanced. Just add a relationship to your taxonomy vocabulary and check the option to require the relationship.
The screen shot below is from the Content view that ships with Drupal 8, which is a great example of how to define a required relationship.