WO2003088131A2

WO2003088131A2 - Method and apparatus for face description and recognition using high-order eigencomponents

Info

Publication number: WO2003088131A2
Application number: PCT/JP2003/004550
Authority: WO
Inventors: Yongsheng Gao; Chak Joo Lee; Sheng Mei Shen; Zhongyang Huang; Takanori Senoh
Original assignee: Matsushita Electric Industrial Co Ltd
Current assignee: Panasonic Holdings Corp
Priority date: 2002-04-12
Filing date: 2003-04-10
Publication date: 2003-10-23
Anticipated expiration: 2004-10-12
Also published as: KR20040101221A; AU2003226455A1; AU2003226455A8; WO2003088131A3; CN1630875A

Abstract

High-order eigencomponents are proposed to describe detailed regional information of that particular facial component. A formula is given to calculate high-order transform matrix for projection. The high-order component features can be used individually to describe a facial component or combined with first -order component features. Since detailed regional identity information can be revealed with high-order component features, the combination of first-order component features and high-order component features of eyes, eyebrows, nose, mouth and outline with different attention weights, based on the significance of identity information of the corresponding area, has better face description capability compared with the first -order eigenface features or combined first -order and high-order eigenface features.

Description

DESCRIPTION

Method and Apparatus for Face Description and Recognition using High-order Eigencomponents

Technical Field

The present invention relates to method and apparatus for face description and recognition using High-order Eigencomponents. The invention can be used in face description and recognition for content-based image retrieval, human face identification and verification for bank, security system and videophone; surveillance and tracking; digital library and Internet multimedia database.

Background Art

Human face perception is an active area in the computer vision community. Face recognition will play an important role in multimedia database search and many other applications. In recent years considerable progress has been made on the problems of face detection and recognition. Different techniques were proposed. Among these, neural networks, elastic template matching, Karhunen-Loeve expansion, algebraic moments and isodensity lines are typical methods. Among these methods, principal component analysis (PCA) or Karhunen-Loeve expansion is an important branch. Eigenface method is derived from PCA and it is convenient to be computed and has consistent accuracy in identification. Prior work showed that the PCA approach dissociates spontaneously between different types of information. The eigenvectors with large eigenvalues capture information that is common to subsets of faces and eigenvectors with small eigenvalues capture information specific to individual face. The studies show that only the information contained by the eigenvectors with large eigenvalues can be generalized to new faces that are not trained.

The advantage of eigenface method is that eigenvectors with large eigenvalues convey information relative to the basic shape and structure of the faces. That means features extracted from eigenvectors with large eigenvalues can be used to describe major characteristics of human faces. However, this is also the weakness of PCA. If we only consider the features extracted from eigenvectors with large eigenvalues, we cannot get the details of faces which corresponds to individual face. If these details of individual face can be described with the common features of human faces, the description of human faces can be more accurate. One drawback of eigenface method is that the contributions of all facial component areas are same. Instead of evenly distributed on the whole face, the identity information mainly locates at the certain facial areas, such as eyes, eyebrows, nose, mouth and outline. The check area contains less identity information and is relatively sensitive to lighting condition changes and facial expression changes. If the identity significances of facial components can be used, the recognition of human face can be more accurate. Disclosure of Invention

Eigenface method is effective to extract common face characteristics such as shape and structure. In order to get details of faces that are lost when eigenvectors with small eigenvalues are truncated, the reconstructed faces with the features from eigenvectors with large eigenvalues should be obtained. With the reconstructed face images, the residue images between original images and reconstructed matrices can be obtained. These residue faces can be looked as high-passed face images, which still contain rich detailed information for individual face. In order to describe these residue faces, eigenface method can be used on these residue faces again. The obtained eigenvectors with large eigenvalues will reveal the common characteristics of residue faces. With this method, high-order eigenvectors with large eigenvalues can be obtained to extract corresponding features. The combination of these features from different order eigenfaces can be used to describe faces effectively.

Similarly, first order and higher order principal components (eigencomponents) of facial components can be obtained to describe the characteristics of the corresponding facial areas. The combination of these features from different order eigencomponents can be used to describe individual facial components efficiently. Finally, human faces can be represented by a combination of different order eigencomponents with different attention weights. For different application fields, different components should have different functionalities (its strongerness or weakness). Different weights should be assigned for that component.

The present invention provides a method to interpret human faces which can be used for image retrieval (query by face example), person identification and verification, surveillance and tracking, and other face recognition applications. In order to describe face characteristics, the concept of high-order eigencomponents is proposed according to our observation and derivation. At first, all face images are normalized to a standard size. Then the vertical location of eyes is calculated and the face is shifted to a suitable place. When ail these pre-processing procedures are finished, the eigencomponents and high- order eigencomponents can be derived from a set of training face images. In order to query a face image in a face database, the features of the image projected with eigencomponents and high-order eigencomponents can be calculated with the selected eigencomponents and high- order eigencomponents. The combination of these features can be used to describe faces. With this description, Euclidean distance can be used for similarity measurement. In order to improve the similarity accuracy, the features should be weighted.

Brief Description of Drawings

Figure 1 shows a flowchart of the procedure for computing first -order feature W^x .

Figure 2 shows a flowchart of the procedures for computing i^-order eigencomponents U' _anc| the

TJ corresponding transform matrix ^»'.

Figure 3 shows a flowchart for the training mode operation.

Figure 4 shows a flowchart for the test mode operation.

Best Mode for Carrying Out the Invention

The present invention gives a method to extract higher order eigencomponent features and represent a face by combining different order component features.

With the normalized face images, the eigencomponents and high-order eigencomponents can be obtained as follows.

First use a preset block for the normalized face images to get the facial component (such as eyes, eyebrows, nose, mouth and outline);

Consider a facial component Φ, which is a one- dimensional vector of raster-scanned facial component, define Ψas the average component:

] M M_i=ι (1)

Every facial component differs from the average component by a vector ⁽¹⁾=Φ,. -Ψ. (2)

The covariance matrix of the data is thus defined as:

Q = A^mA^mT (3) where

Note that Q has dimension whxwh where w is the width of the component and h is the height. The size of this matrix is enormous, but since we only sum up a finite number of component vectors M, the rank of this matrix can not exceed M-1. We note that if v.⁽¹⁾is the eigenvector of A^m A^m (i = 1,2,...,M), then

= λ,^mv,^m where λ are the eigenvalue of A^m A^m , then A^m v^^} are the eigenvectors of

A^mA ^)T as we see by multiplying on the left by A^m in the previous equation:

_Am_AoX_A ^mv,^m

= λ,^mA^mv,^m ( 5 )

Thus, eigenvector v,⁽¹⁾ and eigenvalue 2,⁽¹⁾ are obtained by the following equation.

But A^mTA^m\s only of size M M. So defining w,⁽¹⁾the eigenvector of A^mA^m we have

The eigenvalue λ_t ^m is the variance along the new coordinate space spanned by eigenvectors u_t ^m . From here on we assume that the order of is such that the eigenvalues ⁽¹ are decreasing. The eigenvalues are decreasing in exponential fashion. Therefore we can project a facial component r⁽¹⁾onto only M_λ«M dimensions by computing W^m = {v_k ^m} where w_k ^m = ^(1)Tr⁽¹⁾ and l≤k≤M,. w_k ^m is the /c-th coordinate of r⁽¹⁾in the new coordinate system. In this context, W^m is called first -order component features. The vectors «_fc ^(I)are actually images, and are called first- order eigencomponents. Let ^^ - ^ (8) then _wm _{= u}m^τ _τm_ ₍₉₎

Since U^m s an _jXRmatrix, we cannot get its inverse.

However, we can use its pseudo inverse to approximate its inverse. Let U^m+be the pseudo-inverse of U^mT as indicated below,

U^m+ = Pseudo - Inv(t/^(,)) (10)

then

where f^(I) is the reconstructed matrix from W^mand U^m . Then, the following equation is carried out to obtain a residue component r⁽².

Since a residue facial component vector still contains rich information for the individual component, facial component features should be extracted from the residue component again. Let 4^2> = |τ ^2>r₂ ⁽²\..r_J ²⁾]_I /tf^}be the eigenvalues of ⁽²⁾V²⁾ and be the corresponding eigenvectors of A^m A^{2) . Then ^^(2)r^⁽²⁾ _v ⁽²⁾ _^⁽²⁾ _v ⁽²⁾ Based on above discussion, the eigenvectors of A^l2)Aⁱ²⁾ are w_; ⁽²⁾ = _4⁽²v,⁽²⁾ . Therefore we can project a residue component r⁽²⁾onto only M₂«M dimensions by computing W^m -ψ_k ^{2), where

and l≤k≤M₂. Since u_k ²⁾ are the eigenvectors of the residue component, we call u_k ²⁾ the second -order eigencomponents and w_k ^m the second -order component features.

Let

Eq.(13) can be written as

Let

U₂ = (t^(2)T - U∞^τU^m+U^mT J, (16) we have

Since U₂ is a constant transform matrix and it is just calculated once, it will not affect the efficiency of computation. The facial component can be described with

where 1<M_X ≤M_X. The computational burden does not increase in computing Ω(Φ) compared with only computing component features from eigencomponents U.

The residue components are called second -order residue components and the original components are called first -order residue components.

With the same method, we also can derive third-order, fourth-order ,..., and n^-order eigencomponents. By projecting the residue components of corresponding order, we can get third-order, fourth-order ,..., n^-order component features. With these high-order component features, the similarity of components can be defined as the weighted Euclidean distance between the projections. Fig. 2 illustrates the procedure to compute the ."'-order eigencomponents U⁽ⁱ and the corresponding transform matrix U_t. In the figure, Pseudo_lnv(B) is the function to calculate the pseudo-inverse of matrix B.

The measure of dissimilarity of two faces H₁ and H₂ is defined as a combined distance between various facial component features generated from the projections of eigencomponents (i.e. eigeneyes, eigeneyebrows, eigennoses, eigenmouths and eigenoutlines) and eigenfaces.

M ^mle / M_l ^W-"""

+ ∑ ^O"W,^(I)(Φ₁"^O-)-W_I ^(I)(Φ '|+ ∑ ^{Λ *}W_J ⁽²(Φ,^BO")-W_J ⁽²⁾(^{Φ *}J

If _j=0, the similarity of face images will be measured only with second -order features.

With the method mentioned above, the method for describing a face image can be considered as follows:

1) Scanning the said facial component by using a raster scan starting at the top-left corner of the component window and finishing at the bottom-right corner of the component window into a single dimensional array of pixels;

2) Subtracting the said single dimensional array of pixels with the average component;

3) Multiplying the said subtracted single dimensional array of pixels with the said first-order and high-order eigencomponents; 4) Using the resulting component features as description of the face;

5) Coding the said features into a coded representation. With this method, the human faces can be described effectively and efficiently in space with different attention weights corresponding to the significance of identity information of human face.

Next, an overall operation of the facial feature extraction according to the present invention is described with reference to Figs.3 and 4. The operation includes a training mode operation (steps #22-#31) as shown in Fig.4, and a test mode operation (steps #32-#42) as shown in Fig.3. The training mode operation is carried out first to learn and accumulate many face samples and to obtain coefficients of a first order average.

The training mode starts from step #22 and continues to step #31. The training mode is provided to generate various parameters to be used in the test mode.

In step #22, a plurality of sample face images are input. In step #23, each sample face image is divided into a plurality of facial parts, such as right eye, left eye, right eyebrow, left eyebrow, nose, mouth, facial configuration, and each part is analyzed to obtain basic facial component Φ_;.. The facial component Φ_;. can be weighted according to the significance of identity of human face.

In step #24, the facial components of the same part, such as a nose part, are collected from a plurality of sample face images. The facial component of nose part is referred to as a nose component. The collected facial components of each part are averaged to obtain a first order average facial component Ψ by using equation (1). The same operation is carried out to obtain the first order average facial components for different facial components. The first order average facial component Ψ is used in the test mode operation in step #34. In step #25, the facial component of, for example, nose from each sample face image is subtracted by the first order average facial component Ψ of nose to obtain a vector of nose given by equation (2). The same operation is carried out for each of different facial components.

Steps #22 to #25 taken together is called an analyzing step for analyzing the training face images.

In step #26, equations (4), (5), (6), (7) and (8) are carried out to obtain the first order eigencomponents U^a) . The first order eigencomponents U^m is used in the test mode operation in step #35.

In step #27, an inverse matrix U^m+ is generated by using equation (10). The inverse matrix U^m+ is nearly equal to U^mτ . The inverse matrix H⁽¹⁾⁺ is used in the test mode operation in step #37.

In step #28, the inverse matrix U^m+ is used for obtaining a difference of the facial component using equation (11). Thus, a difference of facial component with respect to the basic facial component of the original facial images as collected in step #22 is obtained. The data produced in step #28 is referred to as a difference facial component.

In step #29, a difference between the difference facial component and the first order average facial component is obtained.

Steps #27 to #29 taken together is called an analyzing step for analyzing the first -order eigencomponent. In step #30, using the difference obtained in step #29, a second order K- L coefficient U^{2) (which is also called second order eigencomponent) is calculated using equations (4), (6), (7), (14) and (16). The second order K-L coefficient U⁽²⁾ is used in the test mode in K-L conversion (step #40). The test mode starts from step #32 and continues to step #42. The test mode is provided to generate first order component feature W^ω and second order component feature W^{2

In step #32, a face to be tested is input.

In step #33, the input face image is divided into a plurality of facial parts, such as right eye, left eye, right eyebrow, left eyebrow, nose, mouth, facial configuration, and each part is analyzed to obtain basic facial component Φ,. . The facial component Φ,. can be weighted according to the significance of identity of human face.

In step #34, the basic facial component is subtracted by the first order average facial component Ψ to obtain a first difference r⁽¹⁾ (also called a first order residue component). The first difference r^ω is applied to step #35 and to step #39.

Steps #32 to #34 taken together is called an analyzing step for analyzing the test face image. In step #35, using the first difference r⁽¹⁾ and the first order eigencomponents H⁽¹⁾ , K-L conversion is carried out by the use of equation (9) to obtain the first order component feature W^m .

In step #36, the first order component feature W ⁽¹⁾ is produced. The first order component feature W^m represents the feature of the test face input in step #32. The first order component feature W^m can be used as information representing the test face, but has a relatively large size of data. A further calculation is carried out to reduce the data size.

In step #37, using the equation (11), K-L inverse conversion is carried out to generate a reconstructed matrix f⁽¹⁾ . In step #38, the reconstructed matrix f ⁽¹⁾ is produced.

In step #39, the equation (12) is carried out to produce a difference between the first difference r⁽¹⁾ and the reconstructed matrix f⁽¹⁾ to obtain a second difference r^{( )} , which is generally referred to as a second residue component r,.⁽²⁾ . Steps #37 to #39 taken together is called an analyzing step for analyzing the first -order component feature.

In step #40, using the second difference r⁽²⁾ and the second order K-L coefficient f/⁽²⁾ , K-L conversion is carried out by equation (17) to generate the second order component feature W ⁽²⁾ . In step #41 , the second order component feature W ⁽²⁾ is produced. The second order component feature W^{{ )} has information representing the test face as entered in the test mode.

It is to be noted that the flowcharts shown in Figs. 3 and 4 can be arranged by a computer connected with a camera for capturing the sample face images and the test face image. It is possible to prepare two sets of arrangements, one set is for the training mode operation, and another set for the test mode operation. Each set includes a computer and a camera. The set for the training mode operation is programmed to carry out steps #22-#30, and the set for the test mode operation is programmed to carry out steps #32-#42. In the set for the test mode operation, a memory is provided to previously store information obtained by the set for the training mode operation, such as the first order average facial component Ψ , the first order eigencomponents U^m , the inverse matrix t/⁽¹⁾⁺ , and the second order eigencomponent U^{2) .

This invention is very effective for describing human faces using component-based features. Since the high-order eigencomponents can be calculated only once with the training components, the high-order component features can be obtained as efficient as first-order component features. However, since detailed regional identity information can be revealed with high- order component features, the combination of first-order component features and high-order component features of eyes, eyebrows, nose, mouth and outline with different attention weights has better face description capability compared with the first -order eigenface features or combined first -order and high-order eigenface features.

This invention is very effective and efficient in describe human faces which can be used in internet multimedia database retrieval, video editing, digital library, surveillance and tracking, and other applications using face recognition and verification broadly.

Claims

1. A method for extracting component feature for face description, comprising: processing a training mode operation, comprising; analyzing a plurality of training face images; calculating a first -order eigencomponent U^m using the analyzed training face images; calculating a second -order eigencomponent £/⁽²⁾ using the analyzed training face images; and processing a test mode operation, comprising; analyzing a test face image; and obtaining a second -order component feature W^{2) for the test face image using the second -order eigencomponents Uⁱ² .

2. A method for extracting component feature for face description, comprising: processing a training mode operation, comprising; analyzing a plurality of training face images to generate a first - order residue component r⁽¹⁾ of the training face; calculating a first -order eigencomponent U^m using the first -order residue component r⁰⁾ of the training face; analyzing the first -order eigencomponent U^m to generate a second -order residue component r⁽²⁾ of the training face; and calculating a second -order eigencomponent H⁽²⁾ using the second -order residue component r⁽²⁾ of the training face; and processing a test mode operation, comprising; analyzing a test face image to generate a first -order residue component r⁽¹⁾ of the test face; obtaining a first -order component feature W^m for the test face image using the first -order eigencomponent U^m and the first -order residue component r⁽¹⁾ of the test face; analyzing the first -order component feature W^m to generate a second -order residue component r⁽²⁾ of the test face; and obtaining a second -order component feature W ² for the test face image using the second -order eigencomponents H⁽²⁾ and the second -order residue component r⁽²⁾ of the test face .

3. The method for extracting component feature as claimed in claim 2, wherein said analyzing a plurality of training face image comprises: dividing each sample face image into facial parts to obtain facial components Φ. of facial parts; averaging the facial component of each facial parts to obtain a first -order average facial component Ψ ; and subtracting the facial component by the first -order average facial component Ψto produce the first -order residue component r⁽¹⁾ of the training face.

4. The method for extracting component feature as claimed in claim 2, wherein said analyzing the first -order eigencomponent comprises: obtaining a reconstructed matrix f⁽¹⁾ ; and subtracting the reconstructed matrix f^ω from the first -order residue component r⁽¹⁾ of the training face to generate the second -order residue component r⁽²⁾ of the training face.

5. The method for extracting component feature as claimed in claim 2, wherein said analyzing a test face image comprises: dividing the test face image into facial parts to obtain facial components

Φ, of facial parts; and subtracting the facial component Φ, by the first -order average facial component Ψ to produce the first -order residue component r⁽¹⁾ of the test face.

6. The method for extracting component feature as claimed in claim 2, wherein said analyzing a test face image comprises: obtaining a reconstructed matrix f⁽¹⁾ ; and subtracting the reconstructed matrix f ⁽¹⁾ from the first -order residue component r⁽¹⁾ of the test face to generate the second -order residue component r⁽²⁾ of the test face.

7. The method for extracting component feature as claimed in claim 3, wherein said facial components Φ_t of facial parts of the training face images can be weighted.

8. The method for extracting component feature as claimed in claim 5, wherein said facial components Φ,. of facial parts of the test face images can be weighted.

9. An apparatus for extracting component feature for face description, comprising: an arrangement operable to processes a training mode operation, comprising; an arrangment operable to analyze a plurality of training face images; an arrangement operable to calculate a first -order eigencomponent U^m using the analyzed training face images; an arrangement operable to calculate a second -order eigencomponent U^{2 using the analyzed training face images; and an arrangement operable to process a test mode operation, comprising; an arrangement operable to analyze a test face image; and an arrangement operable to obtain a second -order component feature W^m for the test face image using the second -order eigencomponents U^{2\

10. An apparatus for extracting component feature for face description, comprising: an arrangement operable to process a training mode operation, comprising; an arrangement operable to analyze a plurality of training face images to generate a first -order residue component r⁽¹⁾ of the training face; an arrangement operable to calculate a first -order eigencomponent U^m using the first -order residue component r⁽¹ of the training face; an arrangement operable to analyze the first -order eigencomponent U^m to generate a second -order residue component r⁽²⁾ of the training face; and an arrangement operable to calculate a second -order eigencomponent U⁽²⁾ using the second -order residue component r⁽²⁾ of the training face; and an arrangement operable to process a test mode operation, comprising; an arrangement operable to analyze a test face image to generate a first -order residue component r⁽¹⁾ of the test face; an arrangement operable to obtain a first -order component feature W^m for the test face image using the first -order eigencomponent U^m and the first -order residue component r⁽ⁱ⁾ of the test face; an arrangement operable to analyze the first -order component feature W^m to generate a second -order residue component r⁽²⁾ of the test face; and an arrangement operable to obtain a second -order component feature W ² for the test face image using the second -order eigencomponents U^{{ )} and the second -order residue component r⁽² of the test face .

11. The apparatus for extracting component feature as claimed in claim 10, wherein said arrangement operable to analyze a plurality of training face images comprises: an arrangement operable to divide each sample face image into facial parts to obtain facial components Φ. of facial parts; an arrangement operable to average the facial component of each facial parts to obtain a first -order average facial component Ψ ; and an arrangement operable to subtract the facial component by the first - order average facial component Ψto produce the first -order residue component r⁽¹⁾ of the training face.

12. The apparatus for extracting component feature as claimed in claim 10, wherein said arrangement operable to analyze the first -order eigencomponent comprises: an arrangement operable to obtain a reconstructed matrix f⁽¹⁾ ; and an arrangement operable to subtract the reconstructed matrix f⁽¹⁾ from the first -order residue component r^(!) of the training face to generate the second -order residue component r⁽²⁾ of the training face.

13. The apparatus for extracting component feature as claimed in claim 10, wherein said arrangement operable to analyze a test face image comprises: an arrangement operable to divid the test face image into facial parts to obtain facial components Φ,. of facial parts; and an arrangement operable to subtract the facial component Φ. by the first -order average facial component Ψto produce the first -order residue component r⁽¹ of the test face.

14. The apparatus for extracting component feature as claimed in claim 10, wherein said arrangement operable to analyze a test face image comprises: an arrangement operable to obtain a reconstructed matrix f⁽¹⁾ ; and an arrangement operable to subtract the reconstructed matrix f^ω from the first -order residue component r⁽¹⁾ of the test face to generate the second - order residue component r⁽²⁾ of the test face.

15. The apparatus for extracting component feature as claimed in claim 11 , wherein said facial components Φ,. of facial parts of the training face images can be weighted.

16. The apparatus for extracting component feature as claimed in claim 12, wherein said facial components Φ,. of facial parts of the test face images can be weighted.

17. An apparatus for extracting component feature for face description, comprising: an arrangement operable to process a training mode operation, comprising; an arrangement operable to analyze a plurality of training face images to generate a first -order residue component r⁽¹⁾ of the training face; an arrangement operable to calculate a first -order eigencomponent U^m using the first -order residue component r⁽¹⁾ of the training face; an arrangement operable to analyze the first -order eigencomponent U^m to generate a second -order residue component r⁽²⁾ of the training face; and an arrangement operable to calculate a second -order eigencomponent U^{2) using the second -order residue component r⁽²⁾ of the training face.

18. An apparatus for extracting component feature for face description, comprising: a memory for storing a first order average facial component Ψ , a first order eigencomponents U^m , an inverse matrix U^m+ , and a second order eigencomponent C7⁽²⁾ ; and an arrangement operable to process a test mode operation, comprising; an arrangement operable to analyze a test face image to generate a first -order residue component r⁽¹⁾ of the test face; an arrangement operable to obtain a first -order component feature W^m for the test face image using the first -order eigencomponent U^m and the first -order residue component r⁽¹⁾ of the test face; an arrangement operable to analyze the first -order component feature W^m to generate a second -order residue component r⁽²⁾ of the test face; and an arrangement operable to obtain a second -order component feature W^{2) for the test face image using the second -order eigencomponents U^{2) and the second -order residue component r⁽²⁾ of the test face .