1. A computer-implemented method for analyzing facial images, the method comprising: accessing data encoding a facial image, the facial image including a face; generating a hierarchical graphical model of the face in the facial image, the hierarchical graphical model including more than nodes from at least one level, each level approximating the face within a contour; applying a bank of filters to the face at a particular node, each filter spanning more than one scale and at least one direction; analyzing filter responses from the bank of filters applied at each direction to obtain a similarity measure that consolidates filter responses from filters applied at more than one scale; generating a vector representation based on the similarity measures to describe the facial image at the particular node; and storing, at a storage device, the generated vector representation from one or more node for computing a distance between two vector representations.