ctools modal form shows ugly access denied
I will try to answer this myself. There is no mentioning of this issue in the CTools project itself but the issue is mentioned at the CTools Auto-modal project https://drupal.org/node/1450232

There is also a solution:


implement hook_page_delivery_callback_alter
check if callback is made by one of your ajax links
if so, check is user has access by using menu_get_item and inspecting the resulting router item
if access is ok, do nothing
if access denied or page not found, render a message 


The solution at CTools Auto-modal renders the message in the CTools modal box. But I preferred to kill the CTools modal and just do nothing:

$commands[] = ctools_modal_command_dismiss();
print ajax_render($commands);
exit();


This way the dreaded drupal ajax error will never appear.