Video compression - why not just sending the changed parts? I'm trying to understand how video compression through dense optical flow works (I'm a newbie with video compression).

I easily understood what optical flow is, the thing I can't understand is why a simple video compression could not just send the parts of a frame marked with a "motion vector" (as you would send only the parts that have moved to or from a position from frame to frame).

I mean: MPEG is way more complex than that, what's the catch with sending parts of a frame that have changed to "stick them over it to get the next one"? Is this a technical flaw (i.e. too much data to be transferred) or is there something else involved that invalidates this simple reasoning?
You could do that, in theory, if you knew "what are the parts" (or objects) frame-to-frame in a video. But:


objects move, change scale, rotate, have color changes, they even can be occluded.
the segmentation is a complicated task in general.


MPEG 4, at least, allows to manipulate objects instead of pixels, see MPEG-4 Natural Video Coding - An overview. If objects and backgrounds are already decoupled, like filmed with a green screen:



then the object contents can be processed separately. Otherwise, what is more important is that, visually, the representation is accurate enough. Thus, a block from some object in a frame  can be cast to another block of a different object in a different frame, as long as the result is satisfactory. Thus, in a lot of cases, the block matching is sufficient, and more importantly, relatively fast to compute.