Why do we need motion vectors in MPEG?
Because that's way more efficient at encoding a lot of typical scenes.
Imagine this: you're filming a complex scene with lots of different objects. The camera shifts ten pixels to the right; with your approach, you'll have to save a relatively large difference for every pixel on average, because the pixels 10 px to the left aren't very similar.
When you first encode the fact that this is just a shift by 10 px, that needs nearly no space at all - and then you'll just need to encode how the scene actually changed.