Solved: Is this a bug - [civicrm.files] variable
UPDATE:
A fix for multisites and Aegir setups will come out in CiviCRM 4.7.21.
See https://issues.civicrm.org/jira/browse/CRM-193



The fix for our problem ended up being somewhat multidimensional.

The patch in https://issues.civicrm.org/jira/browse/CRM-19303 solved the issue of the autogenerated path for the [civicrm.files] variable in 'Settings - upload directories' differing from the URL path generated in 'Settings - Resource URL'.

Though the issue doesn't seem completely solved, the patch has very recently been merged https://github.com/civicrm/civicrm-core/pull/9409

(There is a bit of a rabbit hole that leads to https://github.com/civicrm/civicrm-core/pull/9053).

But this did not totally fix the problem for CKEditor as the CKEditor config file is being placed in the persist directory (instead of persist/contribute) and the persist directory kept returning  a 403 error. Until CRM-19303 is fixed we circumvented it by allowing the vhost to read the persist directory instead of persist/contribute. We temporarily patched hosting_civicrm to do this. 

Aegir was also throwing some file permission errors when creating new sites, and appears to be a different issue, see https://www.drupal.org/node/2791953. We applied the patch which will likely get committed.

We had to enable the Aegir fix_permissions and fix_ownership modules for this, which are experimental, but somewhat necessary here.

For now you still need to manually run a Verify on the site after creating it.

Thanks to Kainuk (https://civicrm.stackexchange.com/users/110/kainuk) for his pointer and to Guillaume Boudrias from praxis.coop (https://www.drupal.org/u/gboudrias) for going down the rabbit holes to fix the problem.