Disallow specific combination of checkbox selections in term reference field?
Take a look at the Field Validation module, it attaches a UI where you can set custom validation rules for each field on a content type. 

I am not sure whether any of the inbuilt options will work for what you want, but you can select the PHP code option and write the rules you require. There is a short reference for the PHP validator here: http://drupal.org/node/1537028