1. A method for processing a video sequence comprised of a plurality of frames, said method comprising: extracting a feature from each of said frames; determining correspondences between said extracted feature from said frames; determining motion in said video sequence based on said determined correspondences, said determining motion using a modified random sample consensus algorithm that selects samples from buckets, iterates a model estimation multiple times including all inliers obtained so far in each iteration, and finds a model that maximizes a data likelihood, wherein a motion hypothesis is derived with a least squares method when the obtained inliers are determined to be less than a number, and the motion hypothesis is derived with a weighted total least squares method otherwise; generating a forward warping matrix and a background warping matrix for each of said frames based on said determined motion; generating a forward warping error and a backward warping error for each of said frames based on said forward warping matrix and said background warping matrix; generating a foreground/background mask for each of said frames based on said forward warping error and said backward warping error; and generating a background mosaic by mapping said frames to a common coordinate system; and extracting foreground information from each of said frames based on said background mosaic.