1. An image encoding apparatus which calculates a prediction error in each predetermined pixel block of an input picture by referring to a pixel in the picture and supplies the prediction error to an encoding unit, thereby encoding the picture, comprising: a storage unit which stores image data of an encoding target picture; a determination unit which determines a size of an encoding target pixel block among three pixel block sizes 4×4, 8×8, and 16×16; a plurality of predicted value calculation units corresponding to all prediction modes of the pixel block size “4×4”; a plurality of prediction error calculation units, each of which calculates, using a predicted value, a prediction error of a predetermined pixel in the pixel block size determined by said determination unit, wherein the plurality of prediction error calculation units includes at least first and second types of prediction error calculation units; wherein the first type prediction error calculation unit calculates, based on a predicted value from a predicted value calculation unit in a prediction mode using one reference pixel value as a predicted value, a prediction error of a pixel in the pixel block size determined by said determination unit and wherein the second type prediction error calculation unit calculates, based on a predicted value from a predicted value calculation unit in a prediction mode for calculating a predicted value based on a plurality of reference pixel values or a predicted value representing one reference value, a prediction error of a pixel in the pixel block size determined by said determination unit; said image encoding apparatus further comprising: an output unit which determines, based on sums of the prediction errors calculated by each of the plurality of prediction error calculation units, a prediction mode which minimizes the sum of the prediction errors, and outputs, to the encoding unit, a prediction error obtained by a prediction error calculation unit corresponding to the determined prediction mode; and a control unit which controls to cause said second type prediction error calculation unit to calculate the prediction error from a plurality of reference pixel values if said determination unit determines the 4×4 pixels as the size of the encoding target pixel block, while said control unit causes said second type prediction calculation unit to calculate the prediction error from one reference pixel value if said determination unit determines the size other than the 4×4 pixels as the size of the encoding target pixel block.