17. A system comprising, a video input source configured to provide a sequence of video frames, each depicting a scene; a processor; and a memory containing a computer vision engine, which when executed by the processor is configured to perform an operation for analyzing an object being tracked in a sequence of video frames, the operation comprising: receiving a representation of the tracked object, as depicted by a current video frame, of the sequence of video frames, evaluating the representation of the tracked object using at least a first classifier and a second classifier, wherein the first classifier is configured to determine a first classification score indicating whether the tracked object depicts an instance of a first classification type, and wherein the second classifier is configured to determine a second classification score indicating whether the tracked object depicts an instance of a second classification type, adding the first classification score to a first rolling average, wherein the first rolling average provides an average of the first classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of video frames, adding the second classification score to a second rolling average, wherein the second rolling average provides an average of the second classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of video frames, wherein the final classification value is determined from the first rolling average and the second rolling average, determining a final classification value for the tracked object in the current video frame, based on the first and second rolling averages, and passing the final classification value for the tracked objects to a machine learning engine configured to identify patterns of behavior engaged in by the tracked object, based at least in part on the final classification value.
17. A system comprising, a video input source configured to provide a sequence of video frames, each depicting a scene; a processor; and a memory containing a computer vision engine, which when executed by the processor is configured to perform an operation for analyzing an object being tracked in a sequence of video frames, the operation comprising: receiving a representation of the tracked object, as depicted by a current video frame, of the sequence of video frames, evaluating the representation of the tracked object using at least a first classifier and a second classifier, wherein the first classifier is configured to determine a first classification score indicating whether the tracked object depicts an instance of a first classification type, and wherein the second classifier is configured to determine a second classification score indicating whether the tracked object depicts an instance of a second classification type, adding the first classification score to a first rolling average, wherein the first rolling average provides an average of the first classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of video frames, adding the second classification score to a second rolling average, wherein the second rolling average provides an average of the second classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of video frames, wherein the final classification value is determined from the first rolling average and the second rolling average, determining a final classification value for the tracked object in the current video frame, based on the first and second rolling averages, and passing the final classification value for the tracked objects to a machine learning engine configured to identify patterns of behavior engaged in by the tracked object, based at least in part on the final classification value. 20. The system of claim 17 , wherein the operation further comprises: training the first classifier using a set of positive training examples depicting instances of the first classification type; training the second classifier using a set positive training examples depicting instances of the second classification type; providing a first set of negative training examples and a second set of negative training examples to the first and second classifiers, respectively, for classification; for each negative training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified negative training examples; for each negative training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified negative training examples.