Inverse of Drupal contextual filters?
In settings of the User ID contextual filter, you can exclude this argument/contextual-filter. Expand the last field-set and you'll see it. isn't it what you need to do ?