1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera, the method comprising: receiving two or more sequences, wherein each sequence corresponds to a segment of a trajectory taken by a respective object through the scene; determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and if the objects interact: mapping each sequence of the two or more sequences to a sequence cluster, and determining a rareness value for the sequence clusters to which the sequences map based on a learned joint probability of the sequence clusters.