Setting site name in installation profile
The solution was to use hook_install_tasks. 

The source of the issue was that Drush was filling in the install configuration form during the installation process. Because the form comes after the profile's hook_install, it will override whatever configuration we write in hook_install.

Using hook_install_tasks will work by defining a new install task for setting our preferred configuration, which Drupal will run after submitting the configuration form. See function install_tasks() in core/includes/install.core.inc, line 716.