Term reference field lists the terms for all the languages
Try selecting Localize. Terms are common for all languages, but their name and description may be localized as the Translation mode, after that remove the terms that you added separately for Spanish language and edit each term that are there in English and translate it to Spanish.