How can I prevent anonymous users from accessing taxonomy pages?
Don't need to force using panels.

Having better solution to set permission for your taxonomy pages by code.

Default taxonomy page permission is 'Access content'.

Override that permission in your own by custom module.

Using hook_menu_alter() to change permission

Sample:

function [MODULENAME]_menu_alter(&amp;$items) {
  $items['taxonomy/term/%']['access arguments'] = array('custom permission');
}


Set that custom permission in your hook_perm().

sample :

function [MODULENAME]_perm() {
  return array('custom permission');
}


Clear your site cache once.

Then Goto admin/user/permissions

Set permission by role.