Is there an API for this?
Are you actually having a problem with user_access or just asking before you're trying it? I'm not really following what exactly is your problem.

Taxonomy Access Control should influence what user_access returns. Or you could just set permissions for "Use contextual links" and let Drupal take care of it with contextual links instead of writing a custom module.