20. The system of claim 14 , further comprising adding a new object type cluster when the micro-feature vector does not correspond to any of the object type clusters for the image data.
14. A system, comprising: a video input source configured to provide image data; a processor; and a memory containing a program, which, when executed on the processor is configured to perform an operation that identifies anomaly object types during classification of image data captured by a video camera, the operation comprising: receiving a micro-feature vector including multiple micro-feature values, each micro-feature value based on at least one pixel-level characteristic of a foreground patch that depicts a foreground object within the image data; classifying the foreground object as depicting a first object type corresponding to a first object type cluster of the object type clusters based on the micro-feature vector; computing a probability density function for the object type clusters; computing a probability density value for the micro-feature vector; evaluating a rareness measure of the micro-feature vector, wherein the rareness measure estimates a likelihood of observing the micro-feature vector, based on the probability density function and the probability density value, and identifying the foreground object as an anomaly object type when the rareness measure is below a specified threshold.