1. A method for processing data describing a scene depicted in a sequence of video frames, the method comprising: receiving input data describing one or more objects detected in the scene, wherein the input data includes at least a classification for each of the one or more objects; identifying one or more primitive events, wherein each primitive event provides a semantic value describing a behavior engaged in by at least one of the objects depicted in the sequence of video frames and wherein each primitive event has an assigned primitive event symbol; generating, for one or more objects, a primitive event symbol stream which includes the primitive event symbols corresponding to the primitive events identified for a respective object; generating, for one or more objects, a phase space symbol stream, wherein the phase space symbol stream describes a trajectory for a respective object through a phase space domain; combining the primitive event symbol stream and the phase space symbol stream for each respective object to form a first vector representation of that object; and passing the first vector representations to a machine learning engine configured to identify patterns of behavior for each object classification from the first vector representation.