How can I handle the user access to specific taxonomy terms?
You may want to try to add your own access checking in your view. A possible way to do so, is to use the Views Access Callback module, which you can yuse in a custom module to make your view perform whatever access check you want. Some more details about this Views Access Callback module (from its project page):

Provides callback-based access control for Views.
Available callback functions are defined by modules using hook_views_access_callbacks() function (in the same way as in hook_perm()), then could be set in in view's &quot;Access restrictions&quot; configuration.

The Community documentation for this module provides some extra details to get you going with this module.