1. A computer-implemented method of creating a motion manifold for determining a similarity between two image patches, the method comprising: identifying semantic regions of videos of a set of digital videos; identifying patch trajectories by tracking patches corresponding to the semantic regions across frames of the videos; creating the motion manifold using the patch trajectories, the motion manifold indicating, for ones of the semantic regions, different visual representations of the semantic region; and storing the motion manifold.