1. A computer-implemented method for analyzing a sequence of video frames depicting a scene captured by a video camera, the method comprising: receiving a set of data inputs derived by a computer vision engine configured to analyze pixels depicting a plurality of foreground objects in the sequence of video frames; and modeling behavior of the foreground objects in the scene by passing the received sensory data inputs to a first cluster layer of a plurality of layers, wherein the plurality of layers alternate between cluster layers and sequence layers, wherein the cluster layers generate clusters of sequences and the sequence layers generate sequences of clusters, and wherein progressively higher levels of the plurality of layers correspond to progressively more complex patterns of behavior engaged in by the foreground objects depicted in the sequence of video frames.