1. A processor-implemented method for tracking an object in a video scene, the method comprising: determining, via at least one processor, at least one pixel-level characteristic for a foreground patch in a first video frame from a plurality of video frames of a video scene, the foreground patch including a set of pixels containing a portion of scene foreground in the first video frame of the video scene; determining, via the at least one processor, kinematics of an object in the video scene based at least in part on the at least one pixel-level characteristic; generating, via the at least one processor, a micro-feature vector based at least in part on the kinematics of the object, the micro-feature vector corresponding to at least one micro-feature value; updating, via the at least one processor, a neural network based at least in part on the micro-feature vector, the neural network including a plurality of clusters, each cluster in the plurality of clusters representing an object type; classifying, via the at least one processor, the object in the first video frame of the video scene as a first object type based on the updated neural network, the first object type being represented by a first cluster in the plurality of clusters; and tracking, via the at least one processor, the object in the video scene based on the micro-feature vector and the first object type.