Views entity reference based on passed in value
Ended up doing a somewhat convoluted hack with a custom module.

1) Used 2 Conditional Filters in the View (A- uid of logged in user, b- uid=0)

2) Used regular filter of the taxonomy type (C).
Since Conditional Filters are ANDed together as well as ANDed with any normal filters, the derived WHERE logic of View is (A AND B) AND C (which would never return results unless the logged in user was user 0, but the query isn't available to anonymous users).

3) Created a custom module and used the following in hook_views_query_alter:
$query->where[0]['type']='OR'
So the WHERE logic of the altered view becomes (A OR B) AND C, which gives me the results I wanted.