Decoding a video signal, how to get frame output to matrix for calculations? I'd like to use python to take a video (eventually a stream) and map each frame to an RGB array of pixels.

My first thought was to use ffmpeg bindings, and in memory (if possible) convert it to png to read each pixel. However, I think that this is redundant to the purpose of the decoder. 

It seems like this may be the most basic entry into video processing, if someone could point me to the library or approach that would accomplish this I would be grateful.
You can do it with OpenCV library. Here is OpenCV link for Python library.