How to use entityQuery to select if a multiple value entityReference field does not contain the id
gapple put me on the right track (at least a solution for my specific problem), and its a combination of his two ideas.

I needed to query for all the nodes that reference my user, then query for all the entities that match my main query. that gives me two arrays of node ids, and if I array_diff the first from the 2nd, I end up with the array of node ids i want. 

so, a separate entityQuery is needed, but its not too computationally expensive in my case, I think. 

$my_contacts = \Drupal::entityQuery('node')
  -&gt;condition('type', 'sc_card')
  -&gt;condition('field_shared_with_user.target_id', $user-&gt;id())
  -&gt;execute();
$public_contacts = \Drupal::entityQuery('node')
  -&gt;condition('type', 'sc_card')
  -&gt;condition('field_public_directory', true)
  -&gt;execute();
$public_not_in_my_contacts = array_diff($public_contacts, $my_contacts);