title: Recommendations for real-time pixel-level analysis of television (TV) video
body: <p>[Note: This is a rewrite of an <a href="https://stackoverflow.com/questions/8378009/which-is-better-for-pixel-level-analysis-of-television-tv-video-opencv-or-ffm">earlier question</a> that was considered inappropriate and closed.]</p>
  
  <p>I need to do some pixel-level analysis of television (TV) video. The exact nature of this analysis is not pertinent, but it basically involves looking at every pixel of every frame of TV video, starting from an MPEG-2 transport stream. The host platform will be server-class, multiprocessor 64-bit Linux machines.</p>
  
  <p>I need a library that can handle the decoding of the transport stream and present me with the image data in real-time. OpenCV and ffmpeg are two libraries that I am considering for this work. OpenCV is appealing because I have heard it has easy to use APIs and rich image analysis support, but I have no experience using it. I have used ffmpeg in the past for extracting video frame data from files for analysis, but it lacks image analysis support (though Intel's IPP can supplement).</p>
  
  <p>In addition to general recommendations for approaches to this problem (excluding the actual image analysis), I have some more specific questions that would help me get started:</p>
  
  <ol>
  <li>Are ffmpeg or OpenCV commonly used in industry as a foundation for real-time
  video analysis, or is there something else I should be looking at?</li>
  <li>Can OpenCV decode video frames in real time, and still leave enough
  CPU left over to do nontrivial image analysis, also in real-time?</li>
  <li>Is sufficient to use ffpmeg for MPEG-2 transport stream decoding, or
  is it preferable to just use an MPEG-2 decoding library directly (and if so, which one)?</li>
  <li>Are there particular pixel formats for the output frames that ffmpeg
  or OpenCV is particularly efficient at producing (like RGB, YUV, or YUV422, etc)?</li>
  </ol>