1. A subject tracking device, comprising: a control device, including at least a CPU, the control device being configured to: compare an input image assuming characteristics quantities corresponding to a plurality of characteristics components, with a template image assuming characteristics quantities corresponding to the plurality of characteristics components, and calculate a first similarity factor indicating a level of similarity between the input image and the template image in correspondence to each of the plurality of characteristics components; normalize the first similarity factors corresponding to the plurality of characteristics components by multiplying the first similarity factors by normalizing values used to equalize the first similarity factors corresponding to the individual characteristic components; and calculate a second similarity factor indicating a level of similarity between the input image and the template image based upon results of normalization of the first similarity factors.