Really confusing composer error output relating to working composer.json/.lock files
Someone else recently reported not being able to update from RC2:

Updating core with Composer doesn&#39;t work

There are two different things, drupal/drupal and drupal/core. You can't mix the two, that's why when you download drupal from drupal.org by hand, you have drupal/drupal and it contains the replace line, as it says that it replaces that.

As suggested there, I'd recommend to use the drupal-project template, I've upgraded many drupal installations to 8.3 with that without a problem.

Note that that has quite a different file structure, specifically it has a web folder that contains the public files.