1. A system for object detection for video sequences using classification-based learning, the system comprising: one or more processors and a non-transitory memory having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of: selecting a fusion method; processing an input video sequence to generate a plurality of detections for each frame in the input video sequence, wherein a detection is a representation of an object candidate within the frame of the input video sequence; fusing the plurality of detections to generate a set of fused detections for each frame using the selected fusion method; sending the set of fused detections to a classification module, a scoring module, and an accept-reject module; wherein the classification module: a) generates a classification score for each fused detection, labeling the fused detection based on a predetermined classification threshold, and passes the classification score to a learning module and the accept-reject module; or b) generates a token indicating that the classification module has abstained from generating a classification score and passes the token to the learning module and the accept-reject module; wherein the scoring module produces a confidence score for each fused detection based on a set of learned parameters from the learning module and the set of fused detections; filtering of the set of fused detections by the accept-reject module based on one of the classification score or the confidence score; and outputting a set of final detections representing an object.