1. An image processing apparatus for detecting an object to be detected in an input image, comprising: an extraction unit configured to extract a reference image region from the input image; a reference color calculation unit configured to calculate a reference color from the reference image region; a feature vector calculation unit configured to (a) calculate color differences between color components of each pixel in a detection window being set in the input image and color components of the reference color, (b) calculate a histogram regarding the color differences in each of local regions, the local regions being defined by dividing an inside of the detection window, and (c) calculate a feature vector including the calculated histograms as elements of the feature vector, each of the elements corresponding to the position of each local region in the detection window; and a discrimination unit configured to discriminate whether or not the object is included in the detection window based on the feature vector.