How can I handle the user access to specific taxonomy terms? There is a node form with a term reference field which's terms list I want to filter based on users/ permissions.

I've tried several modules in which I can grant access either user- or role-based for individual terms. Unfortunately, these settings are ignored by "Views: Filter by an entity reference view" when the views list is built.

So, whats the right way? How do I globally set per-term permissions that are also respected by views listings?

I've thought about changing the term controller but I don't see an access function there. Do I have to use a query alteration? Is there a hook?
You may want to try to add your own access checking in your view. A possible way to do so, is to use the Views Access Callback module, which you can yuse in a custom module to make your view perform whatever access check you want. Some more details about this Views Access Callback module (from its project page):

Provides callback-based access control for Views.
Available callback functions are defined by modules using hook_views_access_callbacks() function (in the same way as in hook_perm()), then could be set in in view's &quot;Access restrictions&quot; configuration.

The Community documentation for this module provides some extra details to get you going with this module.