1. A computer-implemented method of creating a motion manifold for determining a similarity between two image patches, the method comprising: identifying semantic regions of videos of a set of digital videos; for each identified semantic region, identifying patch trajectories by identifying patches corresponding to the semantic region and tracking the patches across consecutive frames of the videos, a patch corresponding to the semantic region including image data for the semantic region; creating a motion manifold indicating different visual representations of a given semantic region by using the patch trajectories identified for the given semantic region, the creating of the motion manifold comprising: clustering the patches into patch clusters according to visual similarity of the patches, each patch cluster representing a subset of the patches having similar visual appearances, the patches of the subset being from one or more patch trajectories, and determining degrees of semantic similarity between pairs of the patch clusters based on degrees to which a first cluster of the pair and a second cluster of the pair have patches from a same patch trajectory; and storing the motion manifold.