hook_custom_theme() works except if a user submits a user form with mismatched passwords. Then they get served the admin theme
hook_custom_theme()'s documentation says :


  The return value of this hook will be used on all pages except those
  which have a valid per-page or per-section theme set via a theme
  callback function in hook_menu(); the themes on those pages can only
  be overridden using hook_menu_alter().


so i guess that what you need here is having your custom rules ina  helper function that you call both in hook_custom_theme() and the theme callback of the error page from hook_menu_alter()