taxonomy vocabulary access
I know, this question is pretty old, but I was looking for a solution by myself.

Maybe someone is looking as well: http://drupal.org/project/vppr
This module provides some new permission settings just for viewing terms which is exactly what I was looking for.