Why doesn't variable_set() work in an installation profile?
variable_set() works fine in installation profiles (standard_install() uses it), it's more likely that the submit handler for your form isn't being run. I seem to remember having to explicitly set the submit handler for one of these extra installation forms:

$form['#submit'][] = 'myprofile_install_mode_form_submit';


Try adding a call to drupal_set_message() in the form submit handler to make sure your code is definitely being run.