How to programmatically submit the file system settings form?
I would not re-create what the form does by hand as @Ayesh does. There is a small chance this form will do more things in a future minor drupal release date (maybe that should never happen, i'm not 100% sure).

What you want to do is look at drupal_form_submit() to programmatically submit a form -- its like visiting the form in a webpage and submitting it without ever opening a browser window. Whatever logic that form does upon submission -- will still happen, eg. create an .htaccess file for you.

You would put the following code into your module install() hook, or you could drush ev this code or make a small php script and run drush php-script myscript.php

Code:

$form_state = array();
$form_state['values']['file_private_path'] = 'sites/default/files/private';
$form_state['values']['op'] = t('Save configuration');
drupal_form_submit('system_file_system_settings', $form_state);


You can then re-submit the form and set the default file serve mode from public to private. Or I would use drush vset or just variable_set somewhere in PHP to set the default file serving mode in drpual. My point is that this is the correct way to programmatically submit a form.