1. A system to perform hiearchical video segmentation, comprising: a processor coupled to a camera; code for defining voxels over a spatio-temporal video; code for grouping into segments contiguous voxels that display similar characteristics including similar appearance or motion; code for determining a trajectory-based feature that complements color and optical flow cues, wherein trajectory cues are probabilistically meaningful histograms combinable for use in a graph-based framework; and code for applying a max-margin module for cue combination that learns a supervised distance metric for region dissimilarity that combines color, flow and trajectory features.