Contextual filter using any term ID
Sorry I wasn't trying to be vague. My view was based on an entity, rather than content, like a node. So I wasn't seeing the "Has taxonomy term id" option in the list of contextual filters. I was only seeing specific taxonomy fields as options. I found that core does not offer that by default for other entities. 

I did find a module that provided the functionality I was seeking:
https://www.drupal.org/project/taxonomy_entity_index

I will also say that this module seems to expect entities to have bundles. If not there is an error. I'm going to submit a patch soon.

Hopefully that will help others. I will take my 2 downvotes and go home now.

(I'll also include this issue link for having the functionality in core: https://www.drupal.org/project/drupal/issues/1040786)