1. An image processing apparatus comprising: a correlation value map generation unit configured to generate a correlation value map by setting a target block having a predetermined size and including a plurality of pixels, on a target image, and obtaining correlation values with a plurality of reference blocks corresponding to a target block set on a reference image different from the target image; a detection unit configured to detect deviation between a reference block having a maximum correlation with the target block from the correlation value map, and the target block, as a motion vector of the target block; and a reliability evaluation unit configured to evaluate reliability of the motion vector by evaluating whether a distribution of correlation values whose correlation are local maximum is linear from the correlation value map.