Disable or hide taxonomy term
The taxonomy term status module is aimed at this use case.

There is an issue for that module regarding views exposed filters at https://www.drupal.org/node/1805882

Looking at that issue it is a little unclear the status of it. Some users say they have it working and others say they do not.
I would try adding your experiences to that issue and someone that has experience with that specific problem may be able to help you get it working.