Remove terms from taxonomy vocabulary for node type
I found an option in the settings for the Views filter called "Limit list to selected items" I checked that and selected all the ones I wanted to show up in the list of terms by the operator settings. That made the ones I didn't want not show up.