1. An image processing apparatus comprising: a correlation value map generation unit configured to generate a correlation value map by setting a target block having a predetermined size and including a plurality of pixels, on a target image, and obtaining correlation values with a plurality of reference blocks corresponding to the target block set on a reference image different from the target image; a detection unit configured to detect deviation between a reference block of the plurality of reference blocks having a maximum correlation with the target block from the correlation value map, and the target block, as a motion vector of the target block; a reliability evaluation unit configured to detect, for each column or for each row of the correlation value map, a point whose correlation is a local maximum of the correlation value map, and to evaluate reliability of the motion vector by evaluating whether a distribution of the detected points, whose correlations are local maximums, is linear on the correlation value map; and a correction unit configured to correct deviation between the target image and the reference image based on the motion vector and the reliability of the motion vector.