How to change "Image style" of an Image field in a given view mode programmatically (as on "Manage display" tab after clicking the gear)?
Something like this should do it:

$entity_type = 'node';
$bundle = 'article';
$field_name = 'field_image';
$view_mode = 'default';

// Get the instance info
$instance_info = field_info_instance($entity_type, $field_name, $bundle);

// Change the image style
$instance_info['display'][$view_mode]['settings']['image_style'] = 'new_image_style';

// Save it
field_update_instance($instance_info);