WO2008064395A1

WO2008064395A1 - Facial feature processing

Info

Publication number: WO2008064395A1
Application number: PCT/AU2007/001169
Authority: WO
Inventors: Brian Lovell; Ting Shan
Original assignee: National ICT Australia Ltd
Current assignee: Data61
Priority date: 2006-08-18
Filing date: 2007-08-17
Publication date: 2008-06-05
Anticipated expiration: 2009-02-18
Also published as: AU2007327540A1

Abstract

This invention concerns a method, software and system for facial feature processing. It may be applied to face recognition as well as to virtual face synthesis. The method comprises the steps of: Capturing an image including a face in any pose. Applying a cascade face detecting algorithm to the image to find the location of the face in the image. Applying an Active Appearance Model (AAM) search to interpret the face located in the image. Estimating the (horizontal and vertical) orientation of the face. Synthesizing a view of the face from any angle, or using Adaptive Principal Component Analysis (APCA) to a synthesized frontal view of the face for recognition.

Description

Title

Facial Feature Processing

Technical Field

This invention concerns a method, software and system for facial feature processing. It may be applied to face recognition as well as to virtual face synthesis.

Background Art

Most face recognition systems only work well under constrained conditions. In particular, the illumination conditions, facial expressions, and head pose must be tightly controlled for good recognition performance. For instance, eigenface-derived face recognition algorithms generally perform well only with front face images.

Disclosure of the Invention

The invention is a method for facial feature processing, comprising the steps of;

Capturing an image including a face in any pose. Applying a cascade face detecting algorithm to the image to find the location of the face in the image.

Applying an Active Appearance Model (AAM) to interpret the face located in the image.

Estimating the (horizontal and vertical) orientation of the face. And, Subsequently synthesizing a view of the face from another angle.

Use of the invention may provide a face recognition technique that is robust to the orientation of the face, both horizontally and vertically, within the captured image. In this case the view of the face that is synthesized is typically the frontal view. A final step in the process is applying Adaptive Principal Component Analysis (APCA) or a similar method, to the frontal view of the face for recognition Experiments show that the approach can achieve improved recognition rates on face images by up to a factor of five compared to standard PCA over a wide range of head poses. The invention may also be applied to the synthesizing of virtual faces (from an image of any face) for game characters. In this case the synthesized view may be of any desired angle.

A Viola- Jones based cascade face detecting algorithm may usefully be used to detect the face in real-time and thus solve the initialization problem for the Active Appearance Model search.

A correlation model may be used to estimate the pose angle in the captured image and to reconstruct frontal model parameters. In this case the correlation model may be applied after the AAM search interprets the face. Following application of the correlation model, the AAM may be used to synthesize the front view of the face.

In another aspect the invention is software configured for performing the method.

In a further aspect the invention is computer hardware programmed with the software.

Brief Description of the Drawings

An example of the invention will now be described with reference to the accompanying drawings, in which:

Fig. 1 is a flow diagram for a process of face detection.

Figs. 2(a)(i), (b)(i), (c)(i) and (d)(i) are face images initialized for Active

Appearance Model (AAM) searches, and Figs. 2(a)(ii), (b)(ii), (c)(ii) and (d)(ii) are the resulting face images marked with the results of failed searches.

Fig. 3 is a block of examples of face images used for training a cascade face detector.

Figs. 4(a)(i), (b)(i), (c)(i) and (d)(i) are images of faces turned to the left. Figs. 4(a)(ii), (b)(ii), (c)(ii) and (d)(ii) are images of faces from the front. And, Figs. 4(a)(iii), (b)(iii), (c)(iii) and (d)(iii) are images of faces turned to the right. In all cases the faces have been labelled with 58 points on the main facial features. Fig. 5(a)(i) is a captured image of a face from the front and Fig. 5(a)(ii) is the frontal face image represented by AAM. Fig. 5 (b)(i) is a captured image of a face from the left; Fig. 5(b)(ii) is the same face image represented by AAM, and Fig. 5 (b)(iii) is a synthesized version of the face from the front.

Fig. 6 is a graph comparing the recognition rate for PCA and APCA on original and synthesized images.

Best Modes of the Invention

Referring first to Fig. 1 the overall process for face recognition will be described. First an image of a face is captured 100, say from a surveillance camera. The face will likely be unknown and will have unknown horizontal and vertical orientation.

The image is then applied to a cascading face detection algorithm 120 that has been suitably trained to locate the face position 130.

The output of the cascading algorithm is applied to an Active Appearance Model (AAM) 140 to precisely interpret the facial features and determine the model shape and texture parameters.

A correlation model is used to estimate the horizontal and vertical orientation of the face, and to reconstruct AAM parameters for a frontal view of the face.

A frontal view of the face is then synthesized 160 using the AAM with the parameters from the correlation model.

Adaptive Principal Component Analysis (APCA) 170, which is insensitive to illumination and expression changes, is then applied to the frontal view to finally recognize the face 180.

This process will now be described in greater detail, and some experimental results will be presented. Training of the Cascade Face Detector 130

The initialization of the Active Appearance Model (AAM) search is a critical problem since the AAM search is a local gradient ascent and some AAM searches have failed in the past due to poor initialization; examples of failed searches can be seen in Fig. 2.

To address this issue a Cascade Face Detector is trained. A face training database includes 4916 hand labeled faces. Also, negative training data were randomly collected from the internet and do not contain human faces. After training, a final face detector has 24 layers with a total of 2913 features. Some training face images can be seen in Fig. 3.

Cascade Filtering 120

The Viola and Jones method [6] combines weak classifiers based on simple binary features which can be computed extremely fast. Simple rectangular Haar-like features are extracted; face and non-face classification is done using a cascade of successively more complex classifiers which discards non-face regions and only sends face-like candidates to the next layer's classifier. Thus it employs a "coarse-to-fine" strategy as described by Fleuret and Geman [8], Each layer's classifier is trained by the AdaBoost learning algorithm. Adaboost is a boosting learning algorithm which can fuse many weak classifiers into a single more powerful classifier.

The cascade face detector finds the location of a human face in an input image and provides a good starting point for the subsequent AAM search.

Active Appearance Model (AAM) 140

Facial Feature Interpretation, such as Active Appearance Models is a powerful tool to describe deformable object images. It demonstrates that a small number of 2D statistical models are sufficient to capture the shape and appearance of a face from any viewpoint. The Active Appearance Model uses principal component analysis (PCA) on the linear subspaces to model both the shape and texture changes of a certain object class. Given a collection of training images for a certain object class where the feature points have been manually marked, a shape and texture can be represented by applying PCA to the sample shape and texture distributions as: x = ^ + P_sc (1) and g = ^~g + P_gc (2) where x is the mean shape, g is the mean texture and P_s , P_g are matrices describing the respective shape and texture variations learned from the training sets. The parameters, c are used to control the shape and texture change.

The AAM search precisely marks the major facial features, such as mouth, eyes, nose and so on.

Experimental Trials

In trials, face image samples were collected from 40 individuals for the rotation model training set. For each person there are 3 images in 3 pose views (left 15°, frontal, right 15°) extracted from the Feret face database [7]. Each of these 120 face images was marked with 58 points around the main features including the eyes, mouth, nose, eyebrows and chin. Some of the labeled training face images can be seen in Fig. 4.

The combined Adaboost-based cascade face detector and AAM search was applied on the rest of the images of the Feret b-series dataset where each person has 7 pose angles, ranging from left 40°, 25°, 15°, 0° to right 15°, 25°, 40°. The AAM search on the face images achieved 95% search accuracy rate. Parameters C₀ , c_c and c_s are learned from the successful AAM search samples.

Correlation Model and Pose Estimation 160

The method of Cootes et al [2] assumes that the model parameter c is related to the viewing angle, θ, approximately by: c = c_o +c_c COs(O¹) + c_s sin(6>) (3)

where c₀ , c_c and c_s are vectors which are learned from the training data by the AAM search. (Here only head turning is considered, head nodding can be dealt with in a similar way). Given a new face image with parameters c, orientation can be estimated as follows.

We first transform equation (3) to:

/ /cos θλ ... c-^c- sineJ ^{( )} Where R-¹ is the left pseudo-inverse of the matrix (c_c \ c_s ), (4) becomes

ysmθj Where (x_a , y_a )^' = R^~x (c - c₀) , then the best estimate of the orientation is tan"¹ (y_a Ix₁₁) -

Frontal-View Synthesis

After the angle θ has been estimated, the model can be used to synthesize new views.

For example to synthesize a frontal view face image, which will be used for face recognition, let c_røbe the residual vector which is not explained by the rotation model, c,-_es = ° - {c_Q + c_s COS(S) + c_s sin(0)) . (6)

To reconstruct at a new angle, α, we simply use the parameters: c(a) - c_o +c_c cos(α) + c_s sin(α) + c_rεs , (7)

here α is 0, so this becomes: C(0) = c_{0 + Cc} + _Crø , (8)

The shape and texture at angle 0°can be calculated by: x(0) =;χ + P_sc(0) (9)

and g(θ) = g + p_εc(θ) (10)

The new frontal face image then can be reconstructed. Fig. 5 shows original faces images from the front and turned, the face images represented by AAM in each case, and a synthesized front view of the turned face represented for comparison. Adaptive Principal Component Analysis (APCA) 170

Adaptive Principal Component Analysis (APCA) inherits merits from both Pattern Classification Algorithms (PCA) and the Fisher Linear Discriminant (FLD), and operates by warping the face subspace according to the within- and between-class co variance. It consists of four steps:

Subspace Projection:

Applying PCA to project face images into the face subspace to generate the m- dimensional feature vectors

Whitening Transformation:

The subspace is whitened according to the eigenvalues of the subspace with a whitening power p cov = diag{^^2p,λ-^2p,...λ-^} (11)

Filtering the Eigenfaces:

Eigen-features are weighted according to the identification-to-variation value ITV with a filtering power q. γ= diag {ITV ^ JTV₂ ⁹ ,... ITV * } (12)

Optimizing the cost function:

Minimize the cost function according to the combination of error rate and the ratio of between-class distance and within-class distance:. ___, τ-iM τ-iκ ■Γ-I d _k (13)

High Pose Angle Face Recognition Results

The APCA face recognition model was trained using the Asian Face Database as in [4]. Then face images were chosen from 46 persons with good AAM search results. Both PCA and APCA were applied to the original face images and to synthesized frontal images respectively for testing. The frontal view images were registered into a gallery and use the high pose angle images for testing. The recognition results are shown in Fig. 6. It can be seen from Fig. 6 that the recognition rates of PCA and APCA on synthesized images is much higher than that of the original high pose angle images. The recognition rate increases by up to 500% for images with view angle of 25°. Yet even for smaller rotation angles less than 15°, the accuracy increases by up to 20%. Note that the recognition performance of APCA is always significantly higher than PCA, which is consistent with the results in [4].

Although the invention has been described with reference to a particular example, it should be appreciated that it could be exemplified in many other forms and in combination with other features not mentioned above. For instance, although the example refers to synthesizing the front face view, any other pose angle could be synthesized.

References

[1] A. Pentland, B.Moghaddam and T.Starner, "View-based and Modular Eigenspaces for Face Recognition", Proc. of ITEE Conference on CVPR 94

[2] T.F.Cootes, K. Walker and C.J.Taylor, "View-based Active Appearance Models", Proc. of The Fourth ITEE International Conference on AFGR 2000

[3] V.Blanz and T.Vetter, "Face Recognition Based on Fitting a 3D Morphable

Model", ITEE Transactions of Pattern Analysis and Machine Intelligence, Volume 25,

Issue 9, Pages: 1063 - 1074, 2003

[4] Shaokang Chen and Brian C. Lovell, "Illumination and Expression Invariant Face Recognition with One Sample Image", ICPR, pp.300-303, 17^th International

Conference on Pattern Recognition (ICPR' 04) -Volume 1, 2004

[5] T.F.Cootes and C.J.Taylor, "Active Appearance Models", IEEE PAMI, Vol.23,

No.6, pp. 681-685, 2001

[6] Paul Viola and Michael Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features", In Proc on CVPR, pp. 511-518, 2001

[7] http://www.itl.nist.gov/iad/humanid/feret/ [last visited 20-Dec-2005]

[8] F.Fleuret and D.Geman, "Coarse-to-Fine Face Detection", International Journal of

Computer Vision, 41:85-107, 2001

[9] Xiujuan Chai, Shiguang Shan and Wen Gao, "Pose normalization for robust face recognition based on statistical affine transformation", Information, Communications and Signal Processing, and the Fourth Pacific Rim Conference on Multimedia, 2003

Claims

1. A method for facial feature processing, comprising the steps of: capturing an image including a face in any pose; applying a cascade face detecting algorithm to the image to find the location of the face in the image; applying an Active Appearance Model (AAM) search to interpret the face located in the image; estimating the (horizontal and vertical) orientation of the face; and. subsequently synthesizing a view of the face from another angle.

2. A method according to claim 1 wherein the view of the face that is synthesized is the frontal view.

3. A method according to claim 2, comprising the further step of applying Adaptive Principal Component Analysis (APCA) or a similar method to the frontal view of the face for recognition.

4. A method according to claim 1, applied to the synthesizing of virtual faces from an image of any face.

5. A method according to any preceding claim, wherein a Viola- Jones based cascade face detecting algorithm is used to detect the face in real-time.

6. A method according to any preceding claim, wherein a correlation model is used to estimate the pose angle in the captured image and to reconstruct frontal model parameters.

7. A method according to claim 5, wherein the correlation model is applied after the AAM interprets the face.

8. A method according to claim 6 or 7, wherein following application of the correlation model the AAM is used to synthesize the front view of the face.

9. Software configured for performing the method according to any preceding claim.

10. Computer hardware programmed with the software according to claim 9.