How to prevent displaying the theme on a custom page callback in hook_menu()
Simply don't return any value, and exit.

In Drupal 7, that can be done by calling drupal_exit(). In Drupal 6, you use code similar to the following one:

function custom_callback() {
  echo 'success';
  module_invoke_all('exit');
  exit();
}


When you call drupal_exit() you don't need to use exit too, as it is already used by the function.

exit() (or drupal_exit()) avoids other hooks are invoked after hook_exit().
Drupal normally calls drupal_page_footer() after the menu callback returns its value; using exit() (or drupal_exit()) avoids that function is called, too. The effect is that the output of the menu callback is not cached. If caching the output is desired then the call to module_invoke_all('exit') and exit(), or drupal_exit() can be removed. As far as the menu callback doesn't return any value, the Drupal page will not be rendered, as it is evident in the Drupal 6 code contained in the index.php file:

$return = menu_execute_active_handler();

// Menu status constants are integers; page content is a string.
if (is_int($return)) {
  switch ($return) {
    case MENU_NOT_FOUND:
      drupal_not_found();
      break;
    case MENU_ACCESS_DENIED:
      drupal_access_denied();
      break;
    case MENU_SITE_OFFLINE:
      drupal_site_offline();
      break;
  }
}
elseif (isset($return)) {
  // Print any value (including an empty string) except NULL or undefined:
  print theme('page', $return);
}

drupal_page_footer();