Programmatically Updating Taxonomy Term
For those of you interested: the bottleneck is in the taxonomy_term_save() function. 

Following the trail, it led me to the field_sql_storage module (in core) which deletes an existing row and then inserts it. It should really use the built in DB Driver functionality and UPSERT instead of deleting and then inserting.

Working issues (for D8 only): 


https://www.drupal.org/node/1800286#comment-9356897
https://www.drupal.org/node/2239071


As of writing this, no patches have been applied yet. Updating large sets of taxonomy terms will lock up your DB.