One contextual filter based on two entity reference fields
Of course I solve this immediately after I post it after hours of trying. If you're in the same situation, it appears the only solution is to use https://www.drupal.org/project/views_contextual_filters_or .

Configure the view as described above. Then additionally, under Query settings, check Contextual filter OR. 

The UI of this module leaves something to be desired but it works so long as all your contextual filters should relate to each other as OR.

The second missing piece was that I was testing this in the view UI using the preview. I was passing the NID of the Location as an argument. What I missed is that because I have two contextual filters I actually need to pass the NID twice like this: "2173/2173". When the view is actually used on the page, the NID is retrieved from the URL and the fact that there are two contextual filters becomes irrelevant.