limit access to some configuration pages (manage display and manage fields)
I would probably go into one of my custom modules and:

invoke a hook_permission() with my new needed permission (administer display and fields) that will be granted only to admin.

Then invoke hook_menu_alter() to alter the access control on the menu path the taxonomy module creates.

This way on install admin will be granted automatically the permission and only the roles granted this permission will be able to access that page.