How to invert a contextual filter which is a taxonomy term ID?
I haven't tested this, but at the bottom of the contextual filter form, under 'More" is a button that reads:
Exclude
If selected, the numbers entered for the filter will be excluded rather than limiting the view.

Have you tried this?  Please follow up with a comment for your results.