What is the best practice to upgrade Drupal core without breaking the Git repo and the production website
It seems from discussion above that your question isn't about fixing your git repo, but about maintaining a Drupal site in git, and how that works with core updates.

I think that standard practice is in fact keeping all files in your repository, including Drupal core. The separation of Drupal code from custom code seems like a nice idea, but I don't think it's necessary, and I don't see what advantages it gives you. It also seems to assume that you will never want to patch core (or have to do it as a part of your deploy), which, while it isn't best practice, is definitely something you want to be able to do if something breaks in production. 
Keeping your commits nice and focused also helps to get this kind of separation.

The solution I've used is to keep all code in the same repository, put as much as possible in Features or other kinds of exports/code/configuration, and maintain a list of patches that need to be applied to out-of-the-box core and contrib.

When you update core, start in your dev environment:


Make sure working directory is clean
Dump latest database (drush sql-dump). Either make a commit with the dump or save it somewhere safe.
Update core (drush up drupal)
Commit all changes. 
Check patches file. If any patches need to be applied, apply them, and make another commit
Run update.php if necessary (already done if you used drush for the update)
Test your dev site. If everything is fine, push code to production. Run drush updb on production.
If there's a problem and you need to revert, either revert or reset your repo (git reset --hard HEAD~2 should do it), or revert the two commits if you want to keep the history. Replace your database with the dump.


The database dump is necessary because you otherwise can't undo changes made to the database by updates. You can also do the updating in a branch, in which case reverting just means checking out master. It's slightly inadvisable if you're working on one dev site because you can't really switch back to master and keep working there in parallel because of the database.

Using this more simple git-side workflow will allow you to use git's full power when you need it without the complication of submodules, etc. If this doesn't seem adequate, could you explain why you need further separation of code?