1. A method, comprising the steps of: receiving a video sequence comprising detection results from one or more detectors, the detection results identifying one or more objects; applying a clustering process to the detection results to identify one or more clusters associated with the one or more objects, wherein the clustering process is applied to the video sequence on a frame-by-frame basis, and wherein applying the clustering process comprises: detecting one or more non-associated detections, wherein the one or more non-associated detections have not been assigned to an existing cluster; applying a non-maximum suppression process to the one or more non-associated detections to generate one or more results; evaluating the one or more results with a confidence score for the one or more non-associated detections, wherein the confidence score comprises a depth-height map and a detection frequency; and instantiating a new cluster or eliminating one or more existing clusters based on the confidence score; determining spatial and temporal information for each of the one or more clusters; associating the one or more clusters to the detection results based on the spatial and temporal information in consecutive frames of the video sequence to generate tracking information; generating one or more target tracks based on the tracking information for the one or more clusters; and consolidating the one or more target tracks to generate refined tracks for the one or more objects; wherein the steps are performed by at least one processor device coupled to a memory.