1. An image processing apparatus that tracks an object present in respective images in a time-series image comprising: an image input unit configured to input the time-series image; a storage unit configured to store object data including object region coordinates which indicate positions of object regions including the object from past images in the time-series image which is images before an input image and a template image for recognizing the object by a template matching; an estimating unit configured to perform the template matching for the input image using the object region coordinate of the object data and the template image and estimate an estimated object region which is a region where the object is present; a similarity generating unit configured to calculate similarities which indicate probabilities of presence of the object using at least one classifier respectively for the estimated object region and a plurality of peripheral regions of the estimated object region; and a determining unit configured to determine the object region and the object region coordinate in the input image on the basis of a distribution range having a higher similarity than a threshold value in a similarity distribution including the respective similarities, generate the template image from a partial image in the object region, and replace and update the object region coordinate and the template image in the past image by the object region coordinate and the template image in the input image.