How to extract pixel-specific time-stream from mp4 video
In video, pixels are not usually encoded independently; you will need to decode full frames. If you are going to run several experiments on a single video, I would suggest that you decode the video once frame-by-frame and write the border pixels into a file in your own raw format for easy access. This should give a manageable file size perhaps of a few gigabytes for a movie-length video.