Admin Block Page Custom Theme
FYI it's a REALLY bad idea to hack core.  Firstly, God knows what could break, and secondly..whatever changes you make will likely be over-written if/when you update core.

To answer your question, you're calling the $custom_theme as you should, but then assigning a Garland to $theme, which isn't used.

Try the following:

function mytheme_init() {
  // Hardcode the administration theme onto the blocks page
  if ($_GET['q'] == 'admin/build/block') {
    global $custom_theme;
    $custom_theme = 'garland';
  }
}


In your custom theme's template.php instead of in block.admin.inc.  Make sure you change mytheme to whatever your custom theme's name is (e.g. my_custom_theme_init or custom_init).

You might also want to look at the Administration Theme module or the Themekey module for admin theme options/theme switching.

Themekey sounds like a pretty good bet for ease of use:


  ThemeKey allows you to define simple or sophisticated theme-switching
  rules which allow automatic selection of a theme depending on current
  path, taxonomy terms, language, node-type, and many, many other
  properties. It can also be easily extended to support additional
  properties exposed by other modules.


EDIT

I forgot, the Admin module also comes with an admin theme option, and a pretty decent admin menu for end users (clients etc.).