1. A method of encoding a video frame comprising: applying a transform to the video frame to obtain transform coefficients; iteratively determining quantized transform coefficients for the transform coefficients by assuming a fixed prediction, applying a soft decision quantization, using the fixed prediction, for fixed quantization step sizes to the transform coefficients to obtain quantized transform coefficients, updating the quantization step sizes using the quantized transform coefficients, and repeating the applying the soft decision quantization for the updated step sizes and the updating the step sizes, until a rate distortion cost associated with the quantized transform coefficients and quantization step sizes is below a predefined threshold; determining a motion compensation in accordance with the iteratively determined quantized transform coefficients and quantization step sizes; and entropy encoding the iteratively quantized transform coefficients and quantization step sizes, wherein the determining a motion compensation includes computing a motion prediction in accordance with a quantized residual frame determined from the iteratively determined quantized transform coefficients and quantization step sizes, iteratively applying the soft decision quantization using the computed motion prediction and updating the step sizes, until a rate distortion cost associated with the quantized transform coefficients encoding is below a predetermined threshold, and repeating the computing the motion prediction and the iteratively applying the soft decision quantization and updating the step size, until an actual rate distortion cost associated with the motion prediction and encoded quantized transform coefficients is below a predetermined rate distortion cost threshold.