I think there are leftovers from previous OS X versions on my HD, how can I find/delete those?
I did a search using the following keywords: mac remove old version

The most promising result described three applications that could work for you.

I suppose that backing up to Time Machine, doing a clean install, and then restoring from Time Machine wouldn't work. The backup would include all the garbage files that you were seeking to be rid of.

Maybe it would be doable to do a clean install, but then only restore selected applications from the backup?