1. A method comprising: identifying a plurality of motion classes for a frame of video data, each motion class having at least one region classified thereto; determining a region to be a poorly classified region forming a set of class hypotheses for the poorly classified region, the set of class hypotheses containing at least one possible motion class; determining a similarity measure for the poorly classified region with respect to a set of past and future video data, the similarity measure indicating a degree of similarity or consistency between a hypothesis for the motion of the poorly classified region and the motion of the corresponding regions in past and future frames; and reclassifying the poorly classified region to one of the at least one possible motion classes according to the similarity measure.