1. A computer-implemented method for identifying a portion of a video as a highlight, comprising: accessing a video; extracting, from the video, a feature set associated with frames of the video; identifying a plurality of video portions of the video, each video portion associated with a subset of frames of the video, and each video portion also associated with a subset of the feature set, wherein the frames of at least one of the video portions overlap with the frames of another one of the video portions; classifying each video portion using a plurality of highlight classifiers and based on the subset of the feature set associated with that video portion to generate a classification result; and identifying one or more video portions of the plurality of video portions as highlights according to the classification results generated by the highlight classifiers.