Git workflow for Composer-based projects
If your goal is to minimize the size of your repository, then you should use Composer, following the pattern described in the project drupal-composer/drupal-project.

One possible workflow would then be:

On dev:


git pull
composer install
drush config-export
git add / commit / push


On stage:


git pull
composer install
drush config-import


On live:


rsync from stage to live
drush config-import


The workflow you propose will not work. The command drush pm-update will not do anything if Drupal is already at the most recent version. drush pm-download is not a good alternative either, as it will overwrite portions of your code base. You could potentially repair this with a git reset --hard or some similar trick, but that would not be very clean.

If you do not wish to use Composer, your best option is to simply commit the core and vendor directories.