Is it misleading to say "holes in semiconductors jump to the other side"?
Both 'hole' and 'electron' are models for what actually happens. It turns out that they can both be modelled as a quantum mechanical 'particle'. These are the best / easiest models to to use to understand semiconductor operation. 

However, 'electron' is rather closer to our limited classical touchy-feely idea of reality than 'hole' is, which is why more of us grok electrons than holes.


  Am I understanding it correct that we just think of electrons leaving behind holes as "holes moving" or do I actually misinterperet the way this works?


I think the biggest problem with your question there is the word 'just'. No, it's not 'just' this, it's much more complicated.