17. A computer system for creating a motion manifold for determining a similarity between two image patches, the system comprising: a computer processor; and a computer-readable storage medium having executable computer program instructions embodied therein that when executed by the computer processor perform actions comprising: identifying semantic regions of videos of a set of digital videos; for each semantic region, identifying patch trajectories by identifying patches corresponding to the semantic region and tracking the patches across consecutive frames of the videos, a patch corresponding to the semantic region including image data for the semantic region; creating a motion manifold indicating different visual representations of a given semantic region by using the patch trajectories identified for the given semantic region, the creating of the motion manifold comprising: clustering the patches into patch clusters according to visual similarity of the patches, each patch cluster representing a subset of the patches having similar visual appearances, the patches of the subset being from one or more patch trajectories, and determining degrees of semantic similarity between pairs of the patch clusters based on degrees to which a first cluster of the pair and a second cluster of the pair have patches from a same patch trajectory; and storing the motion manifold.
17. A computer system for creating a motion manifold for determining a similarity between two image patches, the system comprising: a computer processor; and a computer-readable storage medium having executable computer program instructions embodied therein that when executed by the computer processor perform actions comprising: identifying semantic regions of videos of a set of digital videos; for each semantic region, identifying patch trajectories by identifying patches corresponding to the semantic region and tracking the patches across consecutive frames of the videos, a patch corresponding to the semantic region including image data for the semantic region; creating a motion manifold indicating different visual representations of a given semantic region by using the patch trajectories identified for the given semantic region, the creating of the motion manifold comprising: clustering the patches into patch clusters according to visual similarity of the patches, each patch cluster representing a subset of the patches having similar visual appearances, the patches of the subset being from one or more patch trajectories, and determining degrees of semantic similarity between pairs of the patch clusters based on degrees to which a first cluster of the pair and a second cluster of the pair have patches from a same patch trajectory; and storing the motion manifold. 20. The computer system of claim 17 , wherein creating the motion manifold comprises: forming a cluster graph by creating edges between cluster pairs based on the determined degrees of semantic similarity between the pairs of the patch clusters.