What happens when a compressed video is played back?
In most common compression schemes, there isn't a 1::1 correspondence between "video" frames and "file" (or stream) frames **. Any particular video frame may require the contents of several file frames to become a complete viewable image. So a player will read and buffer as many file frames as needed to reconstruct the video frame in question. The player may buffer much more, but it's not required to.

For example, if some video frame ends up as a P frame when encoded in the file, the player will need to look back to find the previous I frame, decode that, then apply the differences encoded in the intervening P frames to get the complete result.

** EDIT: There is of course a file frame for each video frame, but the point is that you can't usually read one frame from the file and get a complete video frame: that's only true for I frames.