Access to selected taxonomy terms on user profile field
I have achieved this functionality with the following method.

Created a taxonomy term type view which will list only the top level terms.

Added a relationship to view  Taxonomy term: User using Area

Added another relationship Taxonomy term: Parent term

Added a contextual filter on uid for currently logged in user.

This will list only those terms which are selected on user profile. And then display this view as a main menu in the panels.

And then using rules, I return Access denied if a user try to access the taxonomy term page which is not selected on his/her profile.

Hope it will help someone.