Discard views filter on field from referenced taxonomy term when relationship isn't present
Yes, you want to use Grouped Filters in Views. In the View UI, there's an Add/Or, Rearrange option next to the Filter Criteria section.

This allows you to create a complex grouping of filters. To use an example that matches your conditions, here's a setup using Drupal's Standard installation with Basic Page/Article nodes and Tags taxonomy that's exclusive to Article: