Term reference is not available as a contextual filter for a View. Why not?
As mentioned in Niall's comment, you will want to add the taxonomy relationship. In a view that I created recently, I used the Content: Taxonomy terms on node relationship. Once I added it, Content: Has taxonomy term became an available option under Filter criteria. 

That is only one way of doing it. There are a few different taxonomy relationships which serve unique purposes. I'd experiment with them to see what fits your needs.