1. A method of video scene background maintenance for a sequence of video frames, comprising the steps of: aligning the video frames; building a statistical model for each pixel of the aligned video frames; and choosing a background state for each pixel of the aligned video frames.