Core Updates after install with composer create-project drupal/drupal So I installed Drupal with 
composer create-project drupal/drupal &lt;...&gt;

It works fine, but now I'm trying to update core from 8.4.1 to 8.4.4
What's the right way to do that? composer update only updates the modules, not core. 
How do I update core? I don't know that much about composer, but if I edit the composer.json and remove

"replace": {
    "drupal/core": "^8.4"
},


and add

"require": {
    "drupal/core": "^8.4",
    ...
 }


it seems to work.
Is this the right way to do this? If so, why is this not in the default composer.json, that comes with the project?
You do in fact want to remove the drupal/core declaration (as it exists) in the replace section of composer.json, and then add the entry as you noted to the require section.  Then run composer update drupal/core --with-dependencies. 

You can find Drupal's official process for a Core upgrade with composer here: https://www.drupal.org/docs/8/update/update-core-via-composer-option-4