1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera, the method comprising: receiving at least two sequences, wherein each sequence of the at least two sequences corresponds to a segment of a trajectory taken by a respective object through the scene; determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and if the objects interact: mapping each sequence of the at least two sequences to a respective sequence cluster, retrieving, from an ngram trie, a learned joint probability indicating a likelihood of a given sequence cluster pair of a plurality of sequence cluster pairs occurring in the scene, the ngram trie including a plurality of nodes, the given sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the given sequence cluster pair, and determining a rareness value for each sequence cluster pair based on the learned joint probability and a frequency of a most frequently observed sequence cluster pair from the scene, the rareness value given by R ij = 1 - f ij f max , where f ij is a frequency with which sequence cluster pair {C i , C j } has been observed and f max is the frequency of the most frequently observed sequence cluster pair; determining, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value; and upon determining one or more reporting criteria are met based at least on the anomaly temperature, reporting the interaction of the objects.