KR20100081874A - Method and apparatus for user-customized facial expression recognition - Google Patents

Abstract

There is provided a facial expression recognition method and apparatus for extracting feature points that are robust to noise and external factors such as lighting and camera environment while effectively expressing facial expression characteristics of a person, thereby enabling human-independent facial recognition in real time. The facial expression recognition method includes receiving a training image sequence from a user; A expressionless image extracting step of learning a DFEPDM for the received training image sequence and extracting the expressionless image using the learned DFEPDM; Receiving a test image sequence from a user; A D-AAM feature point calculation step of calculating a D-AAM feature point using a difference between AAM) parameters of the expressionless image and the test image sequence; A manifold space projection step of projecting the D-AAM feature points into the learned manifold space to reduce dimensions; And a facial expression recognition step of recognizing the expression of the test image sequence from the D-AAM feature points projected into the manifold space with reference to the gallery sequence. According to the present invention, a differential-AAM feature point can be calculated in real time by finding an expressionless image.

Description

Method and apparatus for user-customized facial expression recognition

The present invention relates to a method and system for recognizing natural facial expressions through image analysis. In particular, the present invention relates to a method for real-time recognition of natural expressions of different people in various lighting and camera environments.

Various techniques for extracting a facial expression of a user from the acquired image have been introduced. In particular, such a technique may be applied to a small camera to increase user convenience by automatically detecting and photographing a smiling face.

As a method of tracking a feature of a subject from an input image, a method of modeling and displaying a pixel change amount between consecutive images as a vector, a method using an active contour, and an image detection method are used. In particular, the image detection method is also used to extract the high frequency components from the image signal to calculate the resolution of the subject, and to drive the imaging device so that the resolution of the subject is maximized.

For a conventional facial expression recognition method, first refer to Korean Patent Application No. 10-2001-0019166. According to this document, a technique is disclosed that can improve the processing speed by increasing the accuracy of feature tracking of a face to be tracked using an extended decision function and reducing the number of comparison operations in the search space. However, this method is a method of recognizing facial expressions for a change from expressionless to object expression, but it is not only difficult to obtain a sequence starting with expressionless and ending with object expression automatically in real time, but also from a specific expression (expression other than expressionless). In the case of turning into a facial expression, there is a limitation that recognition itself is impossible.

In addition, the paper introduced at IEEE International Conference on Fuzzy System held in Seoul in 1999 for facial expression recognition uses fuzzy logic, neural network, and fuzzy neural network. In particular, another method for recognizing facial expressions (national patent 10-2003-0047256) relative to the conventional facial expression recognition method proposes an overall system for facial expression recognition, where 'Soft computing-based Intention reading through the user's mouth for human-friendly human-robot interaction ', Proceedings of SCISamp; See ISIS2002, 23Q3-5, 2002. In addition, H images obtained by HLS color conversion for face recognition / expression recognition are obtained by HLS, respectively, and various image processing techniques are applied to the images. However, these recognition methods have a disadvantage in that they recognize only the extreme facial expressions created artificially with the limitation of showing a high recognition rate only in an environment similar to the learning image.

Therefore, there is an urgent need for a method and apparatus that can efficiently operate in an environment different from an environment in which a learning image is photographed, and can easily perform facial expression recognition customized for each user.

An object of the present invention for solving the above problems is to express facial expression characteristics of the target person effectively to extract facial features independent of humans in real time by extracting feature points that are robust to external factors such as lighting and camera environment and noise. It is to provide a facial expression recognition method that makes it possible.

Another object of the present invention is to provide a facial expression recognition apparatus capable of recognizing not only extreme facial expressions but also natural facial expressions by finally classifying facial expressions using continuous information in a manifold space learned about nonlinear spatial expressions. will be.

One aspect of the present invention for achieving the above object relates to a facial expression recognition method based on the expressionless image of the user, receiving a training image sequence from the user; A facial expression image extraction step of learning a differential facial expression probability density model (DFEPDM) on a received training image sequence and extracting an expressionless image using the learned facial expression probability density model (DFEPDM); Receiving a test image sequence from a user; A D-AAM feature point calculation step of calculating a differential-AAM (D-AAM) feature point using the difference between the Active Appearance Model (AAM) parameters of the expressionless image and the test image sequence; A manifold space projection step of projecting the D-AAM feature points into the learned manifold space to reduce dimensions; And a facial expression recognition step of recognizing the expression of the test image sequence from the D-AAM feature points projected into the manifold space with reference to the gallery sequence. In particular, the gallery sequence is characterized using a differential-AAM feature point sequence for a change from expressionless to predetermined desired expression in manifold space.

The expressionless image extraction step included in the facial expression recognition method according to the present invention may include estimating a density function of feature points having a positive / negative direction using a Gaussian kernel; And subtracting the density function in the negative direction from the density function in the positive direction to generate a differential facial expression probability density model (DFEPDM).

Furthermore, the D-AAM feature point calculation step may include extracting an expressionless image in real time from a training image sequence; And calculating a D-AAM feature point from the extracted expressionless image and test image sequences.

Furthermore, projecting the manifold space includes reducing the dimension of the feature points of the facial expression by projecting it into the learned manifold space to easily represent the nonlinearity of the D-AAM feature points for the test image sequence. It features.

One aspect of the present invention for achieving the above object relates to a facial expression recognition device based on the expressionless image of the user, an image receiver for receiving a training image sequence and a test image sequence from the user to analyze the received image sequence And an image processor for recognizing a facial expression of a user. The image processor included in the facial expression recognition apparatus learns a differential facial expression probability density model (DFEPDM) for the received training image sequence, extracts an expressionless image using the learned facial expression probability density model (DFEPDM), Calculate the D-AAM feature points using the differences between the active appearance model (AAM) parameters of the expressionless image and test image sequences, and project the D-AAM feature points into the learned manifold space to reduce dimensions and in manifold space. Generate a gallery sequence using the differential-AAM feature point sequence for the change from no expression to a desired target expression, and recognize the expression of the test image sequence from the D-AAM feature points projected into the manifold space with reference to the gallery sequence. It is characterized in that it is adapted to.

In particular, the image processor included in the facial expression recognition apparatus according to the present invention is further adapted to recognize facial expressions using a sequence-based k-NNS classification algorithm. In addition, the image processor determines the similarity of the two sequences by considering the weight reflecting the distance and time between two adjacent sequences, and recognizes the facial expression of the test image sequence based on the similarity between the test image sequence and the gallery sequence. Characterized in that it is adapted.

According to the present invention, since the differential-AAM feature point is calculated by referring to the expressionless image in real time and referring to the expression-free image, in this process, not only the variation according to a person, but also the variation of lighting, a camera, and the like is removed.

In addition, by using the manifold learning technique, the dimension can be reduced while effectively expressing the nonlinear space that the facial expression feature points constitute.

Furthermore, by adopting the k-NNS method using the directed Hausdorff distance (DHD), using the sequence information and weighting the distance between the feature points close to the facial expression recognition point, the weight is changed from the expressionless expression to the object expression. Not only the change, but also the change from a specific facial expression to a specific facial expression and an input sequence maintained at the specific facial expression can be effectively recognized.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings which illustrate preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part is said to "include" a certain component, it means that it may further include other components, without excluding the other components unless otherwise stated. In addition, the terms "... unit", "... unit", "module", "block", etc. described in the specification mean a unit that processes at least one function or operation, which means hardware, software, or hardware. And software.

1 is a flowchart illustrating a facial expression recognition method 100 according to the present invention.

First, a training image sequence is received from a user (S110). The training image sequence refers to a series of image sequences obtained to extract a user's expressionless image. For example, the training image sequence may be image frames taken for several seconds using a camera device.

An active appearance model (AAM) is one of the modeling techniques for effectively expressing various face images, and is a parameter of a linear model using principal component analysis. However, the active appearance model (AAM) parameter includes not only features of facial expressions but also various external factors such as lighting, white balance, and sharpness at the time of shooting. Therefore, in order to exclude such factors except facial expressions, the present invention does not use the AAM parameter of any image as it is, but the difference of the AAM parameter using the AAM of the expressionless image of the same user. Calculate the value (D-AAM). D-AAM is a shape parameter and appearance parameter An active appearance model (AAM) parameter is a parameter represented by a shape model and an appearance model, which are briefly illustrated in FIG. 3. For example, the D-AAM may be calculated through concatenation of the shape parameter vector and the appearance parameter vector.

When the training image sequences are received, their differential-AAM (D-AAM) specific points are calculated as in Equation 1.

는 시간 t 에서의 입력 이미지의 D-AAM 특징점이며,

는 시간 t에서의 AAM 파라미터이고,

는 참조 얼굴 이미지의 AAM 파라미터이다. In Equation 1

Is the D-AAM feature point of the input image at time t,

Is the AAM parameter at time t,

Is the AAM parameter of the reference face image.

가 필요하다. 본 발명에 의한 얼굴 표정 인식 방법에서는 무표정 이미지를 참조 얼굴 이미지로 사용한다. 무표정 이미지를 참조 이미지를 사용하는 이유는, 다양한 사용자들의 표정의 특징들을 추출하면 이러한 특징점들이 유사한 성질을 가진다는 점 및 무표정한 표정으로부터 다양한 표정을 자유롭게 표현할 수 있기 때문이다. As can be seen from Equation 1, in order to calculate the D-AAM feature point, the AAM parameter of the reference face image

Is needed. In the facial expression recognition method according to the present invention, the expressionless image is used as the reference face image. The reason why the reference image is used for the expressionless image is that extracting the features of the expressions of various users can express various expressions freely from the fact that these feature points have similar properties and the expressionless expression.

Hereinafter, a method for extracting an expressionless image from a series of training image sequences will be described.

In the present invention, a differential facial expression probability density model (DFEPDM) is trained to find an image closest to the expressionless expression (S120). In this case, a positive value can be given to the direction changing from the expressionless image to the specific facial expression, and negative direction can be given to the direction changing from the specific facial expression to the expressionless expression. Without knowing which face image is expressionless, DFEPDM returns high values for the D-AAM feature points (D-AAM feature points in both directions) calculated using the expression as the reference face image, and certain facial expressions (angry, laughter, surprise). ) Can be learned to return a low value for the D-AAM feature point (the D-AAM feature point in the negative direction) calculated using the expression of the reference face image.

과

의 가우시안 커널을 이용한 밀도 함수를 추정한다. 여기서

와

는 양/음의 방향 각각의 가우시 함수의 분산이다. 그러면, 이를 이용하여 차등 얼굴 표정 확률 밀도 모델(DFEPDM)은 다음 수학식 2와 같이 연산된다. For this DFEPDM learning, the positive / negative D-AAM feature points

and

We estimate the density function using Gaussian kernel of. here

Wow

Is the variance of the Gaussian function in each of the positive and negative directions. Then, using this, the differential facial expression probability density model (DFEPDM) is calculated as in Equation 2 below.

For the real-time input sequence, the D-AAM feature point is calculated by calculating each D-AAM feature point as the reference face image and the value in the DFEPDM is calculated. It has the lowest agreement with the D-AAM feature point in the negative direction, and it can be estimated that it is likely to be expressionless. The D-AAM feature point is calculated as shown in Equation 1 using the expressionless face image found in this way.

When the expressionless image is determined as described above, the test image sequence of the user is received again (S130). The test image sequence may also be generated using an imaging device such as a camera as the training image sequence.

Then, the D-AAM feature point is calculated using the difference between the active appearance model (AAM) parameters of the expressionless image and the test image sequence (S140).

In this process, since the D-AAM feature point for the facial expression changes in nonlinear space, it is preferable to reduce the dimension by using the nonlinear model. Therefore, the D-AAM feature point is projected into the learned manifold space (S150). In the present invention, the facial expression space was studied using k-Isomap proposed by Tenenbaum's "A global geometric framework for nonlinear dimensionality reduction" published in Science in 2000, but the present invention is not limited thereto. It is not. In general, manifolds vary from person to person, so the manifold space must be learned individually for each user. However, since the change from the expressionless image to the specific expression is similar between people, the facial expression recognition method according to the present invention can learn a common manifold space.

When the D-AAM feature point is projected in the manifold space as described above, the facial expression is recognized using a sequence-based algorithm (S160). In particular, the present invention may use a k-NNS classification algorithm, which is a conventional k-NN (k-nearest neighbors) classification algorithm that performs a classification operation with only one image statically. Instead of performing, the algorithm is extended as shown in Equation 3 to process dynamic data. k-NNS generates a gallery as shown in Equation 3 in the learning process.

는 무표정에서 특정 표정(무표정, 화남, 웃음, 놀람)으로 변하는 D-AAM 특징점들을 학습된 매니폴드 공간으로 투영한 i번째 시퀀스이며,

는 i번째 시퀀스의 얼굴 표정 클래스를 의미하고,

는 갤러리의 시퀀스의 총수이다. In Equation 3, S means one sequence,

Is the i-th sequence that projects D-AAM feature points that change from expressionless to specific expressions (expression, anger, laughter, surprise) into the learned manifold space,

Means the facial expression class of the i th sequence,

Is the total number of sequences in the gallery.

와 갤러리상의 i번째 참조 시퀀스

사이의 거리를 측정하기 위하여 방향성 하우스드로프 거리 (Directed Hausdorff Distance, DHD)를 다음 수학식 4와 같이 정의한다.The test process of k-NNS in real time is as follows. Input sequence,

Reference sequence in the and galleries

Directed Hausdorff Distance (DHD) is defined as in Equation 4 below to measure the distance between them.

는 방향성에 대응하는 상수이다.

는 상수값

값에 따라서 최근 정보에 가중치를 두는 가중치 인자로서의 역할을 가지며 다음 수학식 5와 같이 계산된다. here

Is a constant corresponding to directionality.

Is a constant value

It has a role as a weighting factor that weights the latest information according to the value and is calculated as in Equation 5 below.

값이 0이면 모든 시퀀스 상의 특징점들 간의 거리들이 같은 가중치를 가지게 되며,

값이 커질수록 표정인식을 하는 시점에 가까운 특징점들간의 거리는 더 높은 가중치를 취하게 된다. j는 X 시퀀스의 i 번째 특징점에 대응하는 Y 시퀀스 상에서의 특징점의 인덱스로서, 다음 수학식 6과 같이 연산된다. In other words,

If the value is 0, the distances between the feature points in all sequences have the same weight.

As the value increases, the distance between feature points close to the point of time of facial recognition becomes higher. j is an index of a feature point on the Y sequence corresponding to the i th feature point of the X sequence and is calculated as in Equation 6 below.

After calculating the distance between the input sequence and the sequences in the gallery using DHD, k adjacent sequences are found and the facial expressions of the input sequence are classified and recognized by the most selected facial expression class using the majority voting method ( S160).

In other words, the facial expression recognition method according to the present invention includes estimating the image closest to the expressionless expression with respect to the input sequence image, and is defined as the difference from the active appearance model (AAM) parameter of the expressionless image to the AAM parameter of the input image. Calculating a differential-AAM feature point, projecting the non-linear structure of the differential-AAM feature point to the facial expression image thus obtained, into a trained manifold space so that it can be well represented, and sequence-based nearest neighbor (k-NN) Recognizing a facial expression using a classification algorithm.

2 is a block diagram conceptually illustrating an image processor included in a facial expression recognition apparatus according to another aspect of the present invention.

The processor 200 may include first and second active appearance model (AAM) parameter extractors 210 and 215, first and second D-AAM feature point extractors 220 and 225, and first and second manifold spaces. The mapping unit 240 and 250, the DFEPDM calculator 230, the gallery sequence generator 260, and the sequence-based classifier 270 are included. The sequence-based classifier 270 includes a DHD calculator 280 and a K-NNS processor 290. In the illustrated image processor 200, the first active appearance model (AAM) parameter extractor 210, the first D-AAM feature point extractor 220, and the first manifold spatial mapping unit 240 are trained images. In connection with processing the sequence, the second active appearance model (AAM) parameter extractor 215, the second D-AAM feature point extractor 225, and the second manifold spatial mapping unit 250 perform a test image sequence. It is involved in processing.

The active appearance model (AAM) parameter extractor 210, the first D-AAM feature point extractor 220, and the DFEPDM calculator 230 learn a differential facial expression probability density model (DFEPDM) for the received training image sequence. do. In particular, the DFEPDM calculator 230 extracts the expressionless image using the learned facial expression probability density model (DFEPDM).

The learned DFEPDM is mapped to the manifold space by the first manifold space mapping unit 240. The gallery sequence generator 260 then generates a gallery sequence using the differential-AAM feature point sequence for the change from the expressionless expression to the desired expression in the manifold space.

As such, when the manifold space and gallery sequence are generated, the second active appearance model (AAM) parameter extractor 215 receives the test image sequence to extract the active appearance model (AAM) parameters. Then, the second D-AAM feature point extractor 225 calculates the D-AAM feature point by using a difference value between the active expression model (AAM) parameters of the already acquired expressionless image and the received test image sequence. The calculated D-AAM feature point is projected into the learned manifold space by the second manifold space mapping unit 250. Then, the dimension of the D-AAM feature points is reduced.

The DHD calculator 280 calculates a DHD using the projected D-AAM feature point and the gallery sequence received from the gallery sequence generator 260. In addition, the calculated DHD is input to the K-NNS processing unit 290. Then, the K-NNS processing unit 290 obtains the nearest k sequences, determines the similarity of the two sequences in consideration of weights reflecting the distance and time between two adjacent sequences, and determines the test image sequence and the gallery sequence. Recognize facial expressions in test image sequences based on similarities between them.

4 illustrates the D-AAM feature points learned as described above. 4A is a D-AAM feature point and density function in the positive direction, FIG. 4B is a D-AAM feature point and density function in the negative direction, and FIG. 4C is a differential facial expression probability density model obtained by Equation 2 (DFEPDM). ).

Although the present invention has been described with reference to the embodiments shown in the drawings, this is merely exemplary, and it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible.

Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

The present invention can be applied to a method and apparatus for easily extracting various expressions of a user's face while reducing the influence of disturbance from the captured image.

1 is a flowchart illustrating a facial expression recognition method according to an aspect of the present invention.

2 is a block diagram conceptually illustrating an image processor included in a facial expression recognition apparatus according to another aspect of the present invention.

3 is a view conceptually showing that an active appearance model (AAM) applied to the present invention is composed of a shape parameter and an appearance parameter.

4A to 4C are diagrams for explaining a differential facial expression probability density model (DFEPDM) applied to find an expressionless image, and FIG. 4A is a differential-AAM feature point calculated using the expressionless expression as a reference expression and using a Gaussian kernel. The estimated density function is shown.

4B shows a differential-AAM feature point calculated using a specific facial expression as a reference facial expression and a density function estimated using the Gaussian kernel.

4C shows a DFEPDM made using the density functions above.

Claims (12)

In the facial expression recognition method based on the user's expressionless image, Receiving a training image sequence from a user; An expressionless image extraction step of learning a differential facial expression probability density model (DFEPDM) on a received training image sequence and extracting an expressionless image using the trained facial expression probability density model (DFEPDM). ; Receiving a test image sequence from a user; A D-AAM feature point calculation step of calculating a differential AAM (D-AAM) feature point using a difference value between active expression model (AAM) parameters of the expressionless image and the test image sequence; A manifold space projection step of projecting the D-AAM feature points into a learned manifold space to reduce dimensions; And A facial expression recognition step of recognizing an expression of the test image sequence from the D-AAM feature points projected into the manifold space with reference to a gallery sequence; And the gallery sequence is generated using a differential-AAM feature point sequence for a change from no expression to a predetermined desired expression in the manifold space. The method of claim 1, wherein the expressionless image extraction step comprises: Estimating a density function of feature points with a positive / negative direction using a Gaussian kernel; And Subtracting the density function in the negative direction from the density function in the positive direction to generate the differential facial expression probability density model (DFEPDM). The method of claim 1, wherein the D-AAM feature point calculation step, Extracting the expressionless image in real time from the training image sequence; And Calculating the D-AAM feature point from the extracted expressionless image and the test image sequences. The method of claim 1, wherein the manifold space projecting step, Reducing the dimension of the feature points of the facial expression by projecting the manifold space trained to easily represent non-linearity of the D-AAM feature points with respect to the test image sequence. Way. The method of claim 1, wherein the facial expression recognition step comprises: The method is performed using a sequence-based k-NNS classification algorithm. The method of claim 5, wherein the facial expression recognition step, Determining the similarity of the two sequences in consideration of weights reflecting the distance and time between two adjacent sequences; And Recognizing a facial expression of the test image sequence based on a similarity between the test image sequence and the gallery sequence. In the facial expression recognition device based on the expressionless image of the user, An image receiver for receiving a training image sequence and a test image sequence from a user; And An image processor for analyzing a received image sequence to recognize a facial expression of a user, wherein the image processor includes: Learning a differential facial expression probability density model (DFEPDM) on the received training image sequence, extracting an expressionless image using the learned facial expression probability density model (DFEPDM), Calculates a D-AAM feature point using a difference between active expression model (AAM) parameters of the expressionless image and the test image sequence, Projecting the D-AAM feature points into a learned manifold space to reduce dimensions, generating a gallery sequence using a differential-AAM feature point sequence for a change from no expression to a desired expression on the manifold space, and And recognize the expression of the test image sequence from the D-AAM feature points projected into the manifold space with reference to the gallery sequence. The method of claim 7, wherein the image processor, The differential face expression probability density model (DFEPDM) is estimated by using a Gaussian kernel to estimate the density function of feature points with positive / negative direction, and subtract the density function in the negative direction from the density function in the positive direction. And is further adapted to extract the expressionless image by generating an expression. The method of claim 7, wherein the image processor, And is further adapted to calculate the D-AAM feature point by extracting the expressionless image in real time from the training image sequence and calculating the D-AAM feature point from the extracted expressionless image and the test image sequences. Facial expression recognition device. The method of claim 7, wherein the image processor, Facial expression recognition, characterized in that it is further adapted to reduce the dimension of the feature points of the facial expression by projecting it into the learned manifold space to easily represent non-linearity of the D-AAM feature point with respect to the test image sequence. Device. The method of claim 7, wherein the image processor, And further adapted to recognize the facial expression using a sequence based k-NNS classification algorithm. The method of claim 11, wherein the image processor, Determining the similarity of the two sequences in consideration of weights reflecting distance and time between two adjacent sequences, and adapting to recognize a facial expression of the test image sequence based on the similarity between the test image sequence and the gallery sequence. Characterized in that the device.

Images