1. A method of estimating motion and occlusion at a current site in a current frame from a video stream having three or more frames, the method comprising: identifying a plurality of motion vector candidates for each of a forward direction and a backward direction, the identification of at least some motion vector candidates performed at a block level, a pixel level, or a combination thereof using the steps of: initializing the motion vector candidate, and updating and refining the motion vector candidate, the updating using a measure of ill-conditioning of a gradient based estimation of the motion vector candidate and the refining using one or more update energies based on interframe difference, spatial motion correlation, and temporal motion correlation; generating a plurality of candidate pairs, each candidate pair in the plurality of candidate pairs having a forward motion vector from the plurality of motion vector candidates for the forward direction and a backward motion vector from the plurality of motion vector candidates for the backward direction and a occlusion state vector having a forward occlusion state and a backward occlusion state; determining an energy for at least some of the plurality of candidate pairs using a processor by jointly evaluating the forward motion vector and the backward motion vector of at least some of the plurality of candidate pairs based on interframe difference, spatial motion correlation, temporal motion correlation, and spatial occlusion correlation; and estimating motion and occlusion for the current site by selecting a candidate vector from the plurality of candidate vectors based on the determined energies.