Abstract
We describe an algorithm for the automatic features detection 
Introduction
Human faces are characterized by both 3D geometrical shape and color appearance. Therefore, to reproduce with high realism a 3D digital model of a face, the 3D shape and the color field have to be acquired. 3D scanners can be used to the purpose of acquiring and registering color and 3D shape information [3] . Apart from their high cost, 3D scanners require that the real human face is available at scanning time. There are many situations where this cannot be the case. For instance, when reconstructing 3D clones of celebrities [2] the 3D shape can be acquired from wax models, but the color field has to be acquired from footage or old color images which are not registered with the 3D shape. Another important case is the reproduction of human face on a daily basis for web applications. In this case, it would be desirable to have a 3D shape model to which a color image could be sampled and applied on frequently and automatically. In general, a minimum of scanning acquisitions is preferable since they are time consuming and the setup is expensive. On the contrary, as image acquisition can be carried out quickly and with low-cost digital photo cameras, the acquisition of color images and their applica- Analyzing the surface local curvature, points on the 3D surfaces can be identified. As eyes, lips and nose all present marked curvature, they can be identified by using proper spatial filters [7] . More difficult is the extraction of the same features from 2D face images such as "tricks" have been widely used e.g. carefully controlling the internal parameters of the camera or using lip-stick for the lips and markers for the other features [7] .
Introduction
In this framework we propose a more robust method, which works o n natural face images. It is based on a novel integration of standard image processing techniques which use both color and shape information.
Methodology
The method we propose works on images of faces' foregrounds. We thus ignore the problem of localizing the faces in more complex scenes [10] [6] . We acquire the images with homogeneous and light-colored background and only little rotations of the head are accepted (quasi-vertical and frontal position). This conditions are common to most of the face analysis systems (e.g. [15], [17] ). The method works through two hierarchical processing modules: the first identifies four sub-images, each tightly containing one of the features of interest (the two eyes, the nose and the lips); the other three modules are specialized in localizing with high accuracy repere points on the lips, the nose and the eyes.
Identification of sub-images
Starting from a face foreground, we want to localize the eyes, the nose and the lips. At this stage the use of templates [4] [ 181 is computationally too expensive.
Identification of sub-images
An alternative solution is to search the parts of interest by their color. The difficulty here is in the definition of proper color ranges. Inter-individual differences and variations in the illumination produce large variation in the color range of the face components, and does not allow the color characterization of them [16] . For these reasons we moved to consider the grey-level images [ Fig. 1 (a) ].
Identification of sub-images
The first consideration to make is that the skin greylevels are lighter than the ones of the features of interest. Thus it is possible to find a threshold which binarize the image extracting all the features [4] . The question now is how to determine the threshold. As we did not make any asto localize the two eyes separately. The process continues applying the projections on the remaining sub-image. The peak we now obtain in the horizontal projection corresponds to the lips, and the last peak corresponds to the nose. The result of this module is shown in figure 2(b).
Identification of sub-images
At this stage, more refined algorithms are applied to the pixels inside the four sub-images to precisely identify the features of interest and locate repere points.
Identification of sub-images
sumption regarding the illumination, it would be desirable to obtain an adaptive threshold based on the distribution of grey-levels in a particular image. This led us to adopt a competitive learning algorithm [ 11 which divides the image in a predetermined number of clusters. In this case we search for three clusters: one represents the background (the lightest), one the skin (the intermediate grey-level), and the third one the features and the remaining dark pixels of the images (for example the hair) [ Fig.l(b) ]. . Finally boxes containing the features of interest are defined iterating horizontal and vertical projections: first we localize the vertical position of the eyes which corresponds to the highest peak in the horizontal projection of the image in figure 2(a). Then, on the band surrounding the peak, we make the vertical projection in order U and of their middle point. To get a more robust estimate, the entire outline is determined. Classical edge detectors (Sobel, Roberts, Canny, Perona) experimentally fail to produce consistent results both in the determination of the outline and of the repere points. Therefore a novel more refined technique has been developed. This is based on the following steps: localization of the lips cut and of the lips lowest point, localization of the lips corners and of the lips outline.
Lips

Recognition of the lips cut and of the lips lowest point
To determine the lips cut we apply the Sobel vertical derivative operator to the mouth sub-image. This gives better results than determining the Sobel gradient considering that lips cut line is approximately horizontal. We then cluster the vertical derivative values into three clusters: one associated to light-to-dark vertical transition, one to dark-to-light and one to no meaningful transition [ Fig.3(a) ]. We then determine the largest dark connected region and we extract, for every x, the upper pixel belonging to it [ Fig.3(b) ]. The line L connecting these pixels well represents the lips cut apart from its extremes: in some cases, the line ends before the corners, in other, it exceeds them. At this point we are able to recognize the lips lowest point as the lower ex-. treme of the largest connected white region under the line L [ Fig.3(b) ]. This point is significant both because it gives the vertical position of the middle point and for the lips outline recognition.
Recognition of the lips corners and of the lips outline
The key idea here is to arrive to the corners by subsequent refinements. We go back to the original color image considering the mouth box only and clustering it into two clusters. We apply the algorithm in the CIE-Luv color space, being the one which has given experimentally the best results consistently, with respect to RGB, CIE-Lab and HSV color spaces. This lead to obtain one cluster, M , which is roughly associated to the lips and the other to the surrounding skin. The outline of the region M is quite precise in describing the upper lip, but it is not suitable to describe the corners and the outline of the lower lip. We then apply a second time the same clustering algorithm to the sub-image which tightly contains the region M . In this case we require four clusters: the darkest one identifies the shadow corresponding to the lips cut and gives a precise information regarding the horizontal position of the corners. Combining this with the information given by the line L, we obtain the correct corners position [ Fig.3(c) ].
Recognition of the lips corners and of the lips outline
Using all the information we have gathered up to now, we are able to define with sufficient precision the lips' outline. The lower lip is identified as the parabola through the corners and the lips lowest point. For the upper lip, we take the upper part of the outline of M , straining it to finish in the found corners [ Fig.3(d) ].
Eyes
Also in this module we proceed with subsequent refinements: first the eyes are located more precisely, eliminating the eyebrows from the eyes sub-images, second pupils and lower and upper extremes of the eyes are recognized, finally the eyes corners and the upper arcs are defined.
Eyes localization
To obtain a more precise localization of the eye, we apply the Sobel horizontal derivative operator to the greylevel image [ Fig.4(a) ]. We can notice that, corresponding to one side of the iris, the values of the derivative are always the highest. We can therefore localize such pixels using a threshold which binarize the image putting at 1 only the 5% of the brightest pixels and locate the interesting sub-image in the corresponding band [ Fig.4(b) 
Pupil and eye extreme points recognition
Once we have better localized the eye, we proceed looking for the pupil. Some authors (e.g. [ 113, [ 151) suggest finding the darkest pixels. However, this approach is suitable only to particular illumination conditions. Instead, in general, we cannot guarantee this: as can be seen in figure 4(b) , a reflex spot makes several pixels internal to the pupil very bright. As we did not want to make any assumption on the illumination, we have resorted to template matching. As we look for the iris, the template has the shape of a circle '.
Pupil and eye extreme points recognition
The correlation is calculated between the template and the binarized image of the eye [ Fig.S(a) ], obtained clustering it ' we observe here that we adopt a circle of ray 8, but this can easily be generalized by taking a ray which length is proportional to the face dimension.
Pupil and eye extreme points recognition
into two clusters. The center of the pupil, P , is localized in correspondence with the maximum value of the correlation.
Pupil and eye extreme points recognition
Then, the lower extreme of the iris is looked for on the vertical axis A passing throw P: it is localized in the position on the axis A corresponding to the lowest pixel belonging to the white pixels of figure 5(a) . It is not possible to look for the upper extreme of the eye on figure 5(a) , because it would exceed the correct position. What we do is to calculate the absolute value of the vertical derivative, Ay, of the eye image and to threshold it keeping the 10% of the pixels with the highest values. The upper extreme of the eye is localized in correspondence to the intersection of the axis A and the region obtained by the thresholding.
Eye corners and upper arc determination
To define the shape of the eye, we apply an edge following algorithm based on the idea of the hysteresis thresholding [ 121. The two thresholds are automatically determined in order to keep the 10% and the 40% of the pixels with the highest values of Ay. The contour C obtained in this way
Eye corners and upper arc determination
[ Fig.S(b) ] well-defines the shape of the eye but it does not identify precisely the corners. This is the reason why we have the necessity to adopt another method to localize the corners. We start from the consideration that the white internal part of the eye is quite evident. In order to emphasize the contrast between it and the surrounding regions, we equalize the grey-level image and then cluster it requiring 3 clusters [ Fig.S(c) ]. The upper border, U , of such white part [ Fig.S(d) ] gives a good indication to detect both the internal corner of the eye and its outline: combining U and the contour C, we determine the parabola which better approximates them. Then we localize the eye internal corner on the parabola at the height of the end of U , and the external corner on the parabola at the height of the end of C. This allows us to find a good approximation of the upper part of the eye outline[FigS(e)].
Nose
Regarding this feature, we are interested in finding the nose tip. Observing numerous faces' images, we concluded that the nose is characterized by two dark regions, corresponding to the nostrils, and a light region, corresponding to the reflex of the light on the nose tip. In order to identify these regions, we applied the clustering algorithm to the nose grey-level sub-image requiring four clusters. The lightest grey-level pixels correspond to the reflex, and the darkest grey-level pixels correspond to the nostrils [ Fig.6 ]. Moreover we observed that the middle point between the nostrils gives a good vertical localization of the nose tip and that the lower extreme of the region corresponding to the reflex gives a good horizontal localization. The intersection of these two axis gives the nose tip [ Fig.6 ]. 
Results and conclusion
We have applied the method described above to a set of 10 images acquired in different illumination conditions. They represent Caucasian women and men frontal faces. We asked to the people to have a neutral expression, keeping the mouth closed and the eyes opened. Moreover we have not dealt with the case of men with beard or mustaches. All the images have been taken at about the same scale. Working on images at different scales would not create any problem since the algorithm assumes a given resolution just for the template matching step, which can be easily extended to be adapted to the resolution of the image. Figure 7 shows some example results.The method allows identifying three repere points on the mouth, one on the nose and four on each eye for a total of twelve repere points. This is far more than the five required to align the 3D mesh with the 2D image. In general all the repere points were identified successfully.
Results and conclusion
In order to evaluate quantitatively the error carried out by the algorithm, we asked to three subjects to locate on the face images the position of the corners of the eyes and of the lips, and the tip of the nose. In case the answers were different, we considered the average value.
Figure 7. Some results
For each repere point the error has been valued as the distance between the coordinates (xa, ya) determined by the algorithm, and the coordinates (x,, g,) located by the subjects. We adopted the d4 metrics2. Regarding the mouth's corners, we obtained no error in 9 images, an error equal to 1 in 2 images and an error equal to 6 in 1 case (right mouth corner in the right-low image of figure 7) .
Figure 7. Some results
The tip of the nose has been determined incorrectly only in 1 case with an error equal to 3. The estimate of the eye internal corners is reasonable: we obtained no error in 8 images, an error lower than 3 in 2 images and an error equal to 5 in 2 images.
Figure 7. Some results
The most critical point is the determination of the eye external corners which results correct only in 2 cases. In the other images the error is lower than 6. Normalizing the error with respect to the eye dimension (width, height), we obtain, in the worst case, an error of 10%. We believe we can conclude that the precision is high for lips and nose,,while for the eyes the method is less accurate.
Figure 7. Some results
As regard the problem of registering the 3D model and the image, this inaccuracy can be faced considering that we have much more points than we need. Therefore each point contribution can be weighed taking into account its reliability. The running time can be certainly reduced developing the algorithm in a compiled language and optimizing the code.

{'paper_id': '15032790', 'title': 'Automatic features detection for overlapping face images on their 3D range models', 's2_url': 'https://api.semanticscholar.org/CorpusID:15032790', 'source': 'S2ORC'}