Do I need all drupal core history in my own repository
Drupal.org suggests a similar approach.  But, I'm not sure I understand the need to keep core's history under your own repo either. Or any code you yourself didn't create or amend.  

Why not just have your repo at /sites/?  And have your ignore file leave out anything you haven't touched, including unmodified modules from drupal.org (and all the stuff in /sites/default/files/). Then you're workflow could be to use drush to update core, modules and themes and use git for all your own code.  Use drush rsync for files when working between local, dev, stage and production.

Side note: I'm glad to see this question here because this is something I've been wondering myself and though I don't use the approach I've just suggested, I'm thinking about it.