1. A method to perform multi-human three dimensional (3D) tracking, comprising: for each single view, providing two dimensional (2D) human detection candidates from a camera to a 2D tracking module wherein a Convolutional Neural Network (CNN) generates the 2D human detection candidates; a. independently performing 2D tracking in each 2D tracking module and reporting promising 2D tracking hypotheses to a 3D tracking module; b. selecting trajectories from the 2D tracking modules to generate 3D tracking hypotheses; and c. determining a difference score between the detection and the trajectory as a weighted sum of appearance, location, blob size, and orientation.