What is the state-of-the-art of using signal-processing for "cleaning-up" videos? TWO PART QUESTION - We've all seen the movies/TV shows where the police/feds/spies use computer software to take a grainy photo and do a "clean-up" to see a better picture and more details. I assume the concept is executed by some sort of uber-power pixel-smoothing or anti-aliasing type of algorithms to fill in the blanks based on deductive processing.

PART 1: How real is this technology in the public/commercial software world? I'm not asking about any speculation on alleged secret gov software or such, I just want to know where we actually are with this concept today? How much is fully automated vs human-assisted. 

PART 2: Assuming there actually is some reality with this technology for photographs the second part of this question is how (if at all) has this been applied to videos? Again the issue of fully automated vs human assisted is of interest here.

At the heart of this post is the ultimate question of how viable is today's software for being able to take an old VHS or DVD recording and process the frames to create a new HD-resolution remaster. Considering that doing this would mean cleaning up tens-of-thousands of frames for even a simple wedding video I am not expecting this technology to be fast of course.





NOTE: Per in-topic and meta discussions I went ahead and cross-posted this in two other suggested SE forums to acquire their viewpoints and expertise on this matter. So far (it is still a little "early in the day" so to speak) I have received some pretty interesting information in 2 of the 3.

When this is all done (when good answers have been selected) I would appreciate a way to merge these for the benefit of all three SE communities:


Computer Graphics: https://computergraphics.stackexchange.com/questions/1880
Signal Processing: What is the state-of-the-art of using signal-processing for &quot;cleaning-up&quot; videos?
Video Production: https://video.stackexchange.com/questions/17363
Entertainment industry greatly exaggerates what video enhancement can do.

Spatial and temporal noise reduction

Noise reduction by Neat Video and MSU Denoiser Filter, which are somewhere at (according to a comparison from 2007) or near state-of-the-art for hard noise, takes advantage of that adjacent frames usually are of the same scene but with independent noise. So noise can be filtered both spatially and temporally. But that kind of processing is in my experience just cosmetic. It does not really reveal things that a human would not already see through the noise, it just makes the video look nicer. Both filters have an option to adjust settings automatically, but because noise reduction seems always a compromise between noise reduction and introduction of artifacts, the final "strength" of noise reduction should be adjusted to taste manually.

Motion estimation and superresolution

MSU Super-Resolution Filter reconstructs a new video at 2x horizontal and vertical resolution. It follows a similar paradigm as the above two filters, but also estimates motion between frames. The original pixelization (and other artifacts that do not follow the motion such as blocking and sensor Moiré) can be reduced by compensating for motion and then filtering temporally. This improves resolution and makes detailed things like pixel-deprived text and faces more clear. Resolution can be increased only if the pixelization is not properly anti-aliased. Artificial pixelization of faces, if not done carefully, could be reduced by the same or similar processing. This is a kind of enhancement that the human visual system is not accustomed to do natively, and can actually reveal hidden useful information.