How do I take out the caching and bandwidth optimizations sections out of the performance window for a user?
You can use the #access property to do this.  You would just need to wrap things up in your own logic.

/**
 * Implements hook_form_FORM_ID_alter().
 */
function MYMODULE_form_system_performance_settings_alter(&amp;$form, $form_state) {
  if (user_cant_access_this_stuff()) {
    $form['caching']['#access'] = FALSE;
    $form['bandwidth_optimization']['#access'] = FALSE;
  }
}


I am not 100% sure on those $form elements (this was taken from something similar I have on a site), so you may need to dpm() the $form.