CN117877081A - Video facial expression recognition method, system, device, processor and storage medium based on facial key point optimization region characteristics - Google Patents

Abstract

The present invention relates to a video facial expression recognition method based on face key point optimized regional features, comprising: (1) collecting a video facial expression data set and performing preprocessing operations on the video; (2) constructing a spatial feature extraction module of a CNN structure and a feature extraction module based on face key points, respectively extracting relevant information from the collected data; (3) enhancing the information of the extracted relevant information; (4) constructing a feature fusion module to enhance the features of an image within a frame; (5) decoding and processing the extracted relevant information and making a decision, and constructing an overall loss function. The present invention also relates to a corresponding system, device, processor and storage medium thereof. The video facial expression recognition method, system, device, processor and storage medium based on face key point optimized regional features of the present invention are adopted to more effectively utilize face key point features and have a better expression recognition effect compared with a baseline model.

Description

Video facial expression recognition method, system, device, processor and storage medium based on face key point optimization regional features

Technical Field

The present invention relates to the field of digital image technology, in particular to the field of computer vision technology, and specifically refers to a video facial expression recognition method, system, device, processor and computer-readable storage medium thereof based on optimizing regional features of facial key points.

Background technique

Facial expression recognition (FER) is an important direction for computers to understand human emotions and an important aspect of human-computer interaction. Facial expression recognition has broad application value and social significance in understanding and improving human-computer interaction, security, robot manufacturing, automation, medical care, communication, and driving. With the development of computer technology and deep learning technology, as well as the open source of large-scale spontaneous expression datasets in natural environments, facial expression recognition has made significant progress in recent years and has become a research hotspot in academia and industry.

With the continuous development of deep convolutional networks, researchers can mine useful information from large-scale expression data, and various CNN-based methods have been applied to facial expressions. However, due to the complexity of expressions and the imbalance of categories in real scene datasets, facial expression recognition is still a difficult task. For example: the invention patent application with application number: CN202311049424.2 discloses the addition of a self-attention module on the basis of the backbone, which enhances the network's feature extraction ability to focus on specific areas. Its design focuses on the expression recognition ability when occluded faces appear in static image data sets in real scenes; the invention patent application with application number: CN202310942697.3 uses a cross-layer multi-scale channel mutual attention learning mechanism, focusing on convolution kernel design and attention design, and shows advantages in static image data sets in real scenes; the invention patent application with application number: CN202310805217.9 designs a facial expression recognition method based on enhanced self-attention Transformer, uses a lighter network IR50 as the backbone, and then uses an enhanced sub-attention module to enhance the features, and finally uses it for final classification. Its focus is on the lightweight of the network. The defects of the above-mentioned public applications are: they do not pay attention to how to utilize the prior knowledge of time series information and facial key points, and they all directly use the feature maps extracted from the image to perform optimized attention operation recognition and classification, but do not take into account the correlation between time series information, facial key point information and video expression recognition in real scenes, which should be the defect problem that needs to be solved urgently.

Summary of the invention

The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a video facial expression recognition method, system, device, processor and computer-readable storage medium based on facial key point optimized regional features that can effectively utilize the correlation information of previous and next frames of the video and the facial key point information.

In order to achieve the above-mentioned object, the video facial expression recognition method, system, device, processor and computer-readable storage medium thereof based on face key point optimization regional features of the present invention are as follows:

The video facial expression recognition method based on face key point optimization regional features is mainly characterized in that the method comprises the following steps:

(1) Collect video facial expression datasets and perform preprocessing operations on the videos;

(2) Constructing a spatial feature extraction module based on the CNN structure and a feature extraction module based on facial key points to extract primary spatial features and local area features of facial key points from the collected data respectively;

(3) Construct a graph convolution module guided by facial key points to enhance the extracted relevant information;

(4) Construct a feature fusion module to enhance the features of the image within a frame;

(5) Use the timing module and classification module to decode and process the extracted relevant information, make decisions, and construct an overall loss function.

Preferably, the step (1) specifically comprises the following steps:

(1.1) Download the DFEW and AFEW datasets from the dataset official website, obtain the cropped face images through video framing and face extractor, and obtain the original video frames with a size of 256×256pt;

(1.2) Using data augmentation on the original video frame sequence, a final 112×112 pt training and test image is constructed.

Preferably, the data enhancement method includes:

Random sampling: continuously sample 2 frames of a video, and then sample backward 8 times in sequence to obtain the image group for training or testing. The preprocessed image group is randomly rotated [-45°, 45°], and then the image group is randomly flipped horizontally, randomly color-dithered, randomly flipped horizontally, randomly Gaussian blurred, and randomly grayed. Finally, the final training and testing images are obtained through rescaling.

Preferably, the step (2) specifically comprises the following steps:

(2.1) Construct the spatial feature extraction module, specifically:

Use the CNN model as the feature extraction network for the facial expression recognition video data frame, and convert the 112×112×3×16 dimensional input as a sample into a set of 2D feature maps Where h and w represent the length and width of the feature map, d represents the dimension of the feature map, and t represents the number of frames in the sample. Construct a triplet attention module to convert the 2D feature map Convert to 2D feature map/> The dimension of the feature map F is consistent with that of the feature map T;

(2.2) Construct the facial key point feature extraction module, specifically:

A facial key point feature extraction network is constructed as a facial key point feature extraction module, which converts the 112×112×3×16 dimensional input sample into a set of facial key point feature maps. Where N represents the number of facial key points, h and w represent the length and width of the feature map respectively; sample 18 fixed facial key points from the feature map A' and convert the feature map A' into a 2D feature heat map/> And the feature heat map is consistent with the length and width of the feature map T in Section (2.1);

(2.3) Based on the constructed spatial feature extraction module and facial key point feature extraction module, primary spatial domain features and local area features of facial key points are extracted from the collected facial expression recognition data set.

Preferably, the step (3) comprises the following steps:

(3.1) Construct the input features of the graph convolution module guided by the facial key points, specifically:

The 2D feature map obtained using the spatial feature extraction module constructed in Section (2.1) and (2.2) 2D feature heat map constructed by the feature extraction module of 18 facial key points before the festival/> Using pooling calculations and vector element multiplication, and after each operation, layer normalization is used to further adjust the output to obtain the input information of the graph convolution module guided by facial key points/> and/> Such as the following formula:

Among them, g is the global pooling operation, ⊙ is the vector element multiplication operation, () is the vector splicing operation, _Ti is the 2D spatial domain feature map of the i-th frame image, vi _,g is the global face key point feature of the i-th frame image, is the feature heat map of the jth key point of the i-th frame image;

(3.2) Construct the face key point guided graph convolution module, specifically:

Using the sampled and enhanced feature heatmap A, the input features constructed in Section (3.1) and the newly created learnable feature matrix/> Use matrix multiplication, activation function, and matrix element multiplication to convert Such as the following formula:

in, is a matrix multiplication operation, ⊙ is a vector element multiplication operation, T is a matrix transpose operation, f is a full-link operation, A is a feature heat map, W _l is a learnable feature matrix, _Wa is a matrix obtained by matrix multiplication and activation calculation of the learnable matrix and W _l , /> It is the feature vector composed of the first 18 facial key point features in _Vi obtained in Section (3.1), ^and _Vig is the last global feature vector in _Vi .

More preferably, the step 4 specifically comprises the following steps:

(4.1) Divide the face into three regions based on the sampled facial key points, and output information of the facial key point-guided graph convolution module constructed in (3.2) according to the different divided regions. Perform feature fusion operation to construct spatial feature information V _spatial that integrates facial key point information, and constrain the fusion operation with a set of cross entropy loss functions, as shown in the following formula:

_LS = L _class (FC(V _spatial ))

Among them, FC is the full-link operation, L _class (·) is the multi-category cross entropy loss function, and _LS represents the classification loss contributed by the spatial component.

More preferably, the step 5 specifically comprises the following steps:

(5.1) Construct the input of the timing module, specifically:

Construct the fusion feature V _spatial of category feature, position encoding and face key point guidance (3.2), and the temporal module input feature composed of the three Where N is the number of concatenated features, d is the dimension of the feature, and the formula is as follows:

z ⁰ = [x _class ; V _spatial ] + E _pos = [x _class ; v ₁ ; v ₂ ; … ; v _N ] + E _pos

Among them, x _class is the category feature, E _pos is the position code, the symbol is the splicing operation, and V _spatial is the spatial feature information of the fused face key point information;

(5.2) Input features First, a linear mapping layer is passed to generate a query matrix A Key matrix/> And a Value matrix/> Then the three matrices are passed into the multi-head self-attention mechanism MHSA to calculate the weight matrix W, as shown in the following formula:

Where, T is the matrix transpose operation, and d is the normalization constant;

(5.3) Weight matrix W and Value matrix Multiply, and after a residual operation and multi-layer MLP processing, get the output that can be used for classification/> The formula is as follows:

s _i =WV _i ; i∈[1,N]

z ^l = W _H [ s ₁ ; s ₂ ; … ; s _N ] ^T + z ^l-1

Among them, T is the matrix transpose operation, W _H is the learnable weight matrix;

(5.4) Stack multiple layers of attention mechanisms and residual operations described in sections (5.2) and (5.3), and at the last layer, Use a classifier module to map the features to the number of categories c through a fully linked network, and get This vector represents the possible probabilities of c categories. Do a softmax operation to get Used to participate in the calculation of the classification loss function with label supervision, the formula is as follows:

_LT = L _class (R)

Among them, _Lclass (·) is the multi-category cross entropy loss function, and _LT represents the classification loss contributed by the time component.

More preferably, the construction of the overall loss function in step 5 specifically includes the following steps:

The importance weight λ is used to balance the classification loss _LS from the spatial component contribution and the classification loss _LT from the spatial component contribution, as follows:

L _total = λ × _LS + (1-λ) × _LT .

The system for video facial expression recognition based on face key point optimized regional features for implementing the above method is mainly characterized in that the system comprises:

The spatial feature extraction module and the feature extraction module based on facial key points are used to extract spatial feature data and facial key point feature data from the relevant data information in the collected video facial expression recognition data set;

A face key point guided graph convolution module, connected to the spatial feature extraction module and the face key point based feature extraction module, for enhancing the facial expression information in the frame with the help of face key points in the form of graph convolution;

A feature fusion module, connected to the face key point guided graph convolution module, is used to compress high-dimensional features and determine the key feature vectors of emotional information according to face area division;

The timing module and the classification module are connected to the feature fusion module, and a timing feature extraction network is constructed using a multi-head self-attention mechanism MHSA and a multi-layer MLP. The classification results are supervised with labels using the classification network and the cross entropy loss function to obtain the final video facial expression recognition results.

The device for realizing video facial expression recognition based on face key point optimization regional features has the following main features: the device comprises:

a processor configured to execute computer-executable instructions;

The memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the above-mentioned method for video facial expression recognition based on optimizing regional features of facial key points are implemented.

The processor for realizing video facial expression recognition based on face key point optimized regional features has the main feature that the processor is configured to execute computer executable instructions, and when the computer executable instructions are executed by the processor, the various steps of the above-mentioned method for video facial expression recognition based on face key point optimized regional features are realized.

The main feature of the computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the above-mentioned video facial expression recognition method based on optimizing regional features of facial key points.

The method, system, device, processor and computer-readable storage medium for video facial expression recognition based on face key point optimized regional features of the present invention are adopted, and the classic CNN model (such as ResNet18) and face key positioning model (such as Dlib algorithm library) are used as spatial feature extraction module and feature extraction module based on face key points. In order to enhance the potential guiding ability of face key point information, the present invention also innovatively introduces a graph convolution module and a feature fusion module guided by face key points, and strengthens the facial expression information in the frame with the help of face key points. The timing module and the classification module are composed of a multi-head self-attention mechanism MHSA, a multi-layer MLP, and a multi-classification fully linked network, and the potential time features in the video are extracted by stacking multi-layer timing networks. The classifier uses importance weights to balance the classification loss contributed by the spatial component and the classification loss contributed by the spatial component to achieve better results. The technical solution is experimentally verified on the AFEW and DFEW data sets, and has a more prominent classification and recognition effect compared with the baseline model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the overall principle architecture of the method for video facial expression recognition based on face key point optimization regional features of the present invention.

FIG2 is a schematic diagram of a spatial feature extraction module, a feature extraction module based on facial key points, and a graph convolution module guided by facial key points of the present invention.

FIG. 3 is a schematic diagram of a feature fusion module in a method for video facial expression recognition based on face key point optimized regional features of the present invention.

Detailed ways

In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.

Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.

Please refer to FIG. 1 , the method for video facial expression recognition based on face key point optimization regional features, wherein the method comprises the following steps:

(1) Collect video facial expression datasets and perform preprocessing operations on the videos;

(2) Constructing a spatial feature extraction module based on the CNN structure and a feature extraction module based on facial key points to extract primary spatial features and local area features of facial key points from the collected data respectively;

(3) Construct a graph convolution module guided by facial key points to enhance the extracted relevant information;

(4) Construct a feature fusion module to enhance the features of the image within a frame;

(5) Use the timing module and classification module to decode and process the extracted relevant information, make decisions, and construct an overall loss function.

As a preferred embodiment of the present invention, the step (1) specifically comprises the following steps:

(1.1) Download the DFEW and AFEW datasets from the dataset official website, obtain the cropped face images through video framing and face extractor, and obtain the original video frames with a size of 256×256pt;

(1.2) Using data augmentation on the original video frame sequence, a final 112×112 pt training and test image is constructed.

As a preferred embodiment of the present invention, the data enhancement method includes:

Random sampling: continuously sample 2 frames of a video, and then sample backward 8 times in sequence to obtain the image group for training or testing. The preprocessed image group is randomly rotated [-45°, 45°], and then the image group is randomly flipped horizontally, randomly color-dithered, randomly flipped horizontally, randomly Gaussian blurred, and randomly grayed. Finally, the final training and testing images are obtained through rescaling.

As a preferred embodiment of the present invention, the step (2) specifically comprises the following steps:

(2.1) The spatial feature extraction module is constructed in the following manner:

The spatial feature extraction module is constructed using the CNN model to extract spatial feature information from the input samples and calculate a set of 2D feature maps representing preliminary spatial feature information. The module constructs a triplet attention module and calculates a 2D feature map representing spatial enhanced feature information/> The dimension of the feature map F is consistent with that of the feature map T;

(2.2) The facial key point feature extraction module is constructed in the following manner:

The module uses the face key point extraction algorithm to extract a set of face key point feature maps from the input multi-frame images Then, 18 fixed facial key points are sampled from the feature map A' to obtain a 2D feature heat map/>

(2.3) Based on the constructed spatial feature extraction module and facial key point feature extraction module, primary spatial domain features and local area features of facial key points are extracted from the collected facial expression recognition data set.

In practical applications, the above step (2) is specifically as follows:

Step 2.1: Build the feature encoder module:

Use the CNN model as the feature extraction network for the facial expression recognition video data frame, and convert the 112×112×3×16 dimensional input as a sample into a set of 2D feature maps Among them, h and w represent the length and width of the feature map respectively, d represents the dimension of the feature map, and t represents the number of frames in the sample; construct a triplet attention module to convert the 2D feature map Convert to 2D feature map/> The dimension of the feature map F is consistent with that of the feature map T;

Step 2.2: Build the label encoder module:

A facial key point feature extraction network is constructed as a facial key point feature extraction module, which converts the 112×112×3×16 dimensional input sample into a set of facial key point feature maps. Where N represents the number of facial key points, h and w represent the length and width of the feature map respectively; sample 18 fixed facial key points from the feature map A' and convert the feature map A' into a 2D feature heat map/> And the feature heat map is consistent with the length and width of the feature map T in Section (2.1).

As a preferred embodiment of the present invention, the step (3) comprises the following steps:

(3.1) Construct the face key point guided graph convolution module input features, and use the 2D feature map obtained by the spatial feature extraction module constructed in section (2.1) and (2.2) 2D feature heat map constructed by the feature extraction module of 18 facial key points before the festival/> Using pooling calculation and vector element multiplication, we get the input information of the graph convolution module for guiding facial key points/> and/> Such as the following formula:

Among them, g is the global pooling operation, ⊙ is the vector element multiplication operation, () is the vector splicing operation, _Ti is the 2D spatial domain feature map of the i-th frame image, vi _,g is the global face key point feature of the i-th frame image, is the feature heat map of the jth key point of the i-th frame image;

(3.2) Construct the face key point guided graph convolution module: use the sampled and feature enhanced feature heat map A, input feature and learnable feature matrix/> Use matrix multiplication, activation function, matrix element multiplication operation to convert it/> Such as the following formula:

in, is a matrix multiplication operation, ⊙ is a vector element multiplication operation, T is a matrix transpose operation, f is a full-link operation, A is a feature heat map, W _l is a learnable feature matrix, _Wa is a matrix obtained by matrix multiplication and activation calculation of the learnable matrix and W _l , /> It is the feature vector composed of the first 18 facial key point features in _Vi obtained in Section (3.1), ^and _Vig is the last global feature vector in _Vi .

In practical applications, the above step (3) is specifically as follows:

Step 3.1: Construct the face key point guided graph convolution module input features:

The input of the graph convolution module consists of two parts, one of which is the 2D feature map of the spatial feature extractor. The other part is obtained by global average pooling operation; the other part is input by the 2D feature heat map constructed by the face key point feature extraction module/> 2D feature map with spatial feature extractor/> The fusion calculation is obtained by vector element multiplication, and after each operation, layer normalization is used to further adjust the output. The calculation formula for this set of feature vectors is:

Among them, g is the global pooling operation, ⊙ is the vector element multiplication operation, and () is the vector concatenation operation;

Step 3.2: Construct the face key point guided graph convolution module:

Using the sampled and enhanced feature heatmap A, the input feature and the new learnable feature matrix As input, we build a graph convolution module, and use matrix multiplication, activation function, and matrix element multiplication operations to convert it. Where/> The parameters in are randomly initialized, and two consecutive matrix multiplication operations are used to obtain a scientific weight matrix _Wa , which participates in the calculation of graph convolution, as shown in the following formula:

in, is a matrix multiplication operation, ⊙ is a vector element multiplication operation, and f is a full link operation.

As a preferred embodiment of the present invention, the step (4) specifically comprises the following steps:

(4.1) Divide the face into three regions based on the sampled facial key points, and output information of the facial key point-guided graph convolution module constructed in (3.2) according to the different divided regions. Perform feature fusion operation to construct spatial feature information V _spatial that integrates facial key point information, and constrain the fusion operation with a set of cross entropy loss functions, as shown in the following formula:

_LS = L _class (FC(V _spatial ))

Among them, FC is the full-link operation, L _class (·) is the multi-category cross entropy loss function, and _LS represents the classification loss contributed by the spatial component.

In practical applications, the above step (4) is specifically as follows:

Step 4.1: Construct feature fusion module:

The facial area is divided into three parts: left eye, right eye, and lip. The three parts correspond to some feature vectors respectively. After obtaining the features of these three parts, pooling is used to output information of the graph convolution module. Perform feature fusion operation to obtain spatial feature information V _spatial that integrates facial key point information within a frame, and constrain the fusion operation with a set of cross entropy loss functions, such as the following formula:

_LS = L _class (FC(V _spatial ))

Where FC is the full-link operation, L _class (·) is the multi-category cross entropy loss function, and _LS represents the classification loss contributed by the spatial component.

As a preferred embodiment of the present invention, the step (5) is specifically as follows:

(5.1) Construct the input of the temporal module: construct the fusion feature V _spatial guided by category feature, position encoding and facial key points, and the temporal module input feature composed of the three Where N is the number of concatenated features, d is the dimension of the feature, and the formula is as follows:

z ⁰ = [x _class ; V _spatial ] + E _pos = [x _class ; v ₁ ; v ₂ ; … ; v _N ] + E _pos

Among them, x _class is the category feature, E _pos is the position code, and the symbol is the concatenation operation.

(5.2) Input features First, a linear mapping layer is passed to generate a query matrix A Key matrix/> And a Value matrix/> Then the three matrices are passed into the multi-head self-attention mechanism MHSA to calculate the weight matrix W, as shown in the following formula:

Where, T is the matrix transpose operation, and d is the normalization constant;

(5.3) Weight matrix W and Value matrix Multiply, and after a residual operation and multi-layer MLP processing, get the output that can be used for classification/> The formula is as follows:

s _i =WV _i ; i∈[1,N]

z ^l = W _H [ s ₁ ; s ₂ ; … ; s _N ] ^T + z ^l-1

Among them, T is the matrix transpose operation, W _H is the learnable weight matrix;

(5.4) Stack multiple layers of attention mechanisms and residual operations described in sections (5.2) and (5.3), and at the last layer, Use a classifier module to map the features to the number of categories c through a fully linked network, and get This vector represents the possible probabilities of c categories. Do a softmax operation to get Used to participate in the calculation of the classification loss function with label supervision, the formula is as follows:

_LT = L _class (R)

Among them, _Lclass (·) is the multi-category cross entropy loss function, and _LT represents the classification loss contributed by the time component.

In practical applications, the above step (5) is specifically as follows:

Step 5.1: Construct the input of the timing module:

The input of the timing module consists of three parts: category features, position coding, and fusion features V _spatial guided by facial key points. The category features are composed of a randomly initialized vector whose dimension needs to be consistent with the input dimension of the timing module to ensure that they can be directly added. The position coding uses a sine-cosine alternating position coding algorithm to determine the time sequence of the input. The timing module input features composed of the three are Where N is the number of concatenated features, d is the dimension of the feature, and the formula is as follows:

z ⁰ = [x _class ; V _spatial ] + E _pos = [x _class ; v ₁ ; v ₂ ; … ; v _N ] + E _pos

Step 5.2: Construct the attention weight matrix:

The attention weight matrix is learned through self-attention operation, which requires the input features After a linear mapping layer, a query matrix is generated/> A Key matrix/> And a Value matrix The mapping relationship of the three matrices does not share parameters. Then the three matrices are passed into the multi-head self-attention mechanism MHSA to calculate the weight matrix W, which represents the higher influence of certain areas in the feature map, as shown in the following formula:

Where, T is the matrix transpose operation, and d is the normalization constant;

Step 5.3: Build a timing model:

The calculated weight matrix W and Value matrix After multiplication, a residual operation and multi-layer MLP processing, a feature vector with the same dimension as the input feature can be obtained. This feature vector better integrates the time series features and can be used as the input of the next layer of time series network:

s _i =WV _i ; i∈[1,N]

z ^l = W _H [ s ₁ ; s ₂ ; … ; s _N ] ^T + z ^l-1

Among them, T is the matrix transpose operation, W _H is the learnable weight matrix;

Step 5.4: Stacking the Timing Network

The attention mechanism and residual operation described in the stacking multi-layer section, the number of stacked layers affects the network's performance in extracting temporal features, and the last layer has an important effect on the output The dimensions of are the same as the input features. A classifier module is used to map the features to the number of categories c through a fully linked network, and we get /> This vector represents the possible probabilities of c categories. Do a softmax operation to get /> Used to participate in the calculation of the classification loss function with label supervision, the formula is as follows:

_LT = L _class (R)

Among them, L _class (·) is the multi-class cross entropy loss function, and _LT represents the classification loss contributed by the time component. The importance weight λ is used to balance the classification loss _LS contributed by the spatial component and the classification loss _LT contributed by the spatial component. The formula is as follows:

L _total = λ × _LS + (1-λ) × _LT .

The system for video facial expression recognition based on face key point optimization regional features using the above method, wherein the system comprises:

The spatial feature extraction module and the feature extraction module based on facial key points are used to extract spatial feature data and facial key point feature data from the relevant data information in the collected video facial expression recognition data set;

The face key point guided graph convolution module is connected to the spatial feature extraction module and the feature extraction module based on face key points, and is used to enhance the facial expression information in the frame with the help of face key points in the form of graph convolution.

The feature fusion module is connected to the face key point guided graph convolution module, and is used to compress high-dimensional features and determine the key feature vectors of emotional information according to face area division.

The timing module and the classification module are connected to the feature fusion module, and a timing feature extraction network is constructed using a multi-head self-attention mechanism MHSA and a multi-layer MLP. The classification results are supervised with labels using the classification network and the cross entropy loss function to obtain the final video facial expression recognition results.

In a specific embodiment of the present invention, the classification and identification method using the present technical solution is tested as follows:

(1) Experimental Dataset

The present invention uses video facial expression recognition datasets AFEW and DFEW for experimental verification. The AFEW dataset has been an evaluation dataset for emotion recognition in The Wild Challenge (EmotiW) from 2013 to 2019, during which the dataset was updated. The dataset is a temporal multimodal database containing audio and video data. The samples are labeled with seven emotion base classes: happiness, surprise, anger, disgust, fear, sadness, and neutrality. The AFEW dataset provides 1,809 videos, including 773 in the training set, 383 in the validation set, and 653 in the test set. In order to ensure the rigor of the data, the three sets of videos are not repeated, and even the identities of the people in the videos are not the same. The DFEW dataset contains samples from more than 1,500 realistic high-definition movies, covering a variety of topics, and realistically reflects people's facial movements in various environments. The dataset contains 16,372 video samples and also contains 7 emotion base classes. The samples of the dataset are evenly divided into five non-repeating parts. One of the parts is selected as the test set during the test, and the other parts are used as training sets.

(2) Training process

The training images were scaled to 112×112 pt, and data enhancement methods such as random rotation, grayscale transformation, and random folding were used. The Adam optimizer was used, the initial learning rate was set to le-3, the learning rate was decayed with a fixed round, the batch was set to 64, and a total of 80 rounds of training were used.

(3) Test results

In this embodiment, training is performed on the AFEW and DFEW datasets respectively, and then the performance is tested on the validation sets of the two datasets, and Accuracy (Acc.) is selected as the algorithm evaluation index. The experimental results are shown in Table 1.

Table 1 Performance comparison of models trained on AFEW and DFEW datasets (%)

As can be seen from Table 1, the performance of this embodiment on the AFEW and DFEW datasets is better than that of the previous methods, and can achieve the best performance when detecting on the two datasets.

The device for realizing video facial expression recognition based on face key point optimization regional features, wherein the device comprises:

a processor configured to execute computer-executable instructions;

The memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the various steps of the above-mentioned method for video facial expression recognition based on optimizing regional features of facial key points are implemented.

The processor for realizing video facial expression recognition based on face key point optimized regional features, wherein the processor is configured to execute computer executable instructions, and when the computer executable instructions are executed by the processor, the various steps of the above-mentioned video facial expression recognition method based on face key point optimized regional features are realized.

The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the various steps of the above-mentioned method for video facial expression recognition based on optimizing regional features of facial key points.

Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

The method, system, device, processor and computer-readable storage medium for video facial expression recognition based on face key point optimized regional features of the present invention are adopted, and the classic CNN model (such as ResNet18) and face key positioning model (such as Dlib algorithm library) are used as spatial feature extraction module and feature extraction module based on face key points. In order to enhance the potential guiding ability of face key point information, the present invention also innovatively introduces a graph convolution module and a feature fusion module guided by face key points, and strengthens the facial expression information in the frame with the help of face key points. The timing module and the classification module are composed of a multi-head self-attention mechanism MHSA, a multi-layer MLP, and a multi-classification fully linked network, and the potential time features in the video are extracted by stacking multi-layer timing networks. The classifier uses importance weights to balance the classification loss contributed by the spatial component and the classification loss contributed by the spatial component to achieve better results. The technical solution is experimentally verified on the AFEW and DFEW data sets, and has a more prominent classification and recognition effect compared with the baseline model.

In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims (11)

1. A video facial expression recognition method based on face key point optimization regional features, characterized in that the method comprises the following steps: (1) Collect video facial expression datasets and perform preprocessing operations on the videos; (2) Constructing a spatial feature extraction module based on the CNN structure and a feature extraction module based on facial key points to extract primary spatial features and local area features of facial key points from the collected data respectively; (3) Construct a graph convolution module guided by facial key points to enhance the extracted relevant information; (4) Construct a feature fusion module to enhance the features of the image within a frame; (5) Use the timing module and classification module to decode and process the extracted relevant information, make decisions, and construct an overall loss function. 2. The video facial expression recognition method based on face key point optimization regional features according to claim 1, characterized in that the step (1) specifically comprises the following steps: (1.1) The captured video is subjected to video framing and face extraction to obtain a cropped face image, and an original video frame with a size of 256×256 is obtained; (1.2) Data augmentation and downsampling are used on the original video frame sequence to construct the final 112×112 training and test images; the data augmentation method includes: random sampling, continuously sampling 2 frames of a group of videos, sequentially sampling backward 8 times to obtain the image group for training or testing, randomly rotating the pre-processed image group [-45°, 45°], and then performing random horizontal flipping, random color jittering, random horizontal flipping, random Gaussian blurring and random graying on the image group, and finally obtaining the final training and test images through size scaling operations. 3. The video facial expression recognition method based on face key point optimization regional features according to claim 1, characterized in that the step (2) specifically comprises the following steps: (2.1) Construct the spatial feature extraction module, specifically: Use the CNN model as the feature extraction network for the facial expression recognition video data frame, and convert the 112×112×3×16 dimensional input as a sample into a set of 2D feature maps Where h and w represent the length and width of the feature map, d represents the dimension of the feature map, and t represents the number of frames in the sample. Construct a triplet attention module to convert the 2D feature map Convert to 2D feature map/> The dimension of the feature map F is consistent with that of the feature map T; (2.2) Construct the facial key point feature extraction module, specifically: The 112×112×3×16 dimension input as a sample is extracted and converted into a set of facial key point feature maps Where N represents the number of facial key points, h and w represent the length and width of the feature map respectively; sample 18 fixed facial key points from the feature map A' and convert the feature map A' into a 2D feature heat map/> And the feature heat map is consistent with the length and width of the feature map T in Section (2.1); (2.3) Based on the constructed spatial feature extraction module and facial key point feature extraction module, primary spatial domain features and local area features of facial key points are extracted from the collected facial expression recognition data set. 4. The video facial expression recognition method based on face key point optimized regional features according to claim 1, characterized in that the face key point guided graph convolution module in step (3) comprises the following steps: (3.1) Construct the input features of the graph convolution module guided by the facial key points, specifically: The 2D feature map obtained using the spatial feature extraction module constructed in Section (2.1) and (2.2) 2D feature heat map constructed by the feature extraction module of 18 facial key points before the festival/> Using pooling calculations and vector element multiplication, and after each operation, layer normalization is used to further adjust the output to obtain the input information of the graph convolution module guided by facial key points/> and/> Such as the following formula: Among them, g is the global pooling operation, ⊙ is the vector element multiplication operation, () is the vector splicing operation, _Ti is the 2D spatial domain feature map of the i-th frame image, vi _,g is the global face key point feature of the i-th frame image, is the feature heat map of the jth key point of the i-th frame image; (3.2) Construct the face key point guided graph convolution module, specifically: Use the sampled and enhanced feature heatmap A, and the input features constructed in Section (3.1) And the newly created learnable feature matrix/> Use matrix multiplication, activation function, matrix element multiplication operation to convert it/> Such as the following formula: in, is a matrix multiplication operation, ⊙ is a vector element multiplication operation, T is a matrix transpose operation, f is a full-link operation, A is a feature heat map, W _l is a learnable feature matrix, _Wa is a matrix obtained by matrix multiplication and activation calculation of W _l , _Vi ^l is a feature vector composed of the first 18 facial key point features in _Vi obtained in Section (3.1), _{and Vig} ^is the last global feature vector in _Vi . 5. The video facial expression recognition method based on face key point optimization regional features according to claim 4 is characterized in that the face is divided into three regions according to the sampled face key points, and the face key point guided graph convolution module output information constructed in (3.2) is used according to the different divided regions. Perform feature fusion operation to construct spatial feature information V _spatial that integrates facial key point information, and constrain the fusion operation with a set of cross entropy loss functions, as shown in the following formula: _LS = L _class (FC(V _spatial )) Among them, FC is the full-link operation, L _class (·) is the multi-category cross entropy loss function, and _LS represents the classification loss contributed by the spatial component. 6. The video facial expression recognition method based on face key point optimization regional features according to claim 5, characterized in that the timing module and the classification module comprise the following steps: (5.1) Construct the input of the timing module, specifically: Construct the fusion feature V _spatial of category feature, position encoding and face key point guidance (3.2), and the temporal module input feature composed of the three Where N is the number of concatenated features, d is the dimension of the feature, and the formula is as follows: z ⁰ = [x _Class ; V _spatial ] + E _pos = [x _class ; v ₁ ; v ₂ ; … ; v _N ] + E _pos Among them, x _class is the category feature, E _pos is the position code, the symbol is the splicing operation, and V _spatial is the spatial feature information of the fused face key point information; (5.2) Input features First, a linear mapping layer is passed to generate a query matrix/> A Key matrix/> And a Value matrix/> Then the three matrices are passed into the multi-head self-attention mechanism MHSA to calculate the weight matrix W, as shown in the following formula: Where, T is the matrix transpose operation, and d is the normalization constant; (5.3) Weight matrix W and Value matrix Multiply, and after a residual operation and multi-layer MLP processing, get the output that can be used for classification/> The formula is as follows: s _i =WV _i ; i∈[1,N] z ^l = W _H [ s ₁ ; s ₂ ; … ; s _N ] ^T + z ^l-1 Among them, T is the matrix transpose operation, W _H is the learnable weight matrix; (5.4) Stack multiple layers of attention mechanisms and residual operations described in sections (5.2) and (5.3), and at the last layer, Use a classifier module to map the features to the number of categories c through a fully linked network, and get/> This vector represents the possible probabilities of c categories. Do a softmax operation to get /> Used to participate in the calculation of the classification loss function with label supervision, the formula is as follows: L _t = L _class (R) Among them, _Lclass (·) is the multi-category cross entropy loss function, and _LT represents the classification loss contributed by the time component. 7. The video facial expression recognition method based on face key point optimization regional features according to claim 1 is characterized in that the overall loss function constructed in step (5) is specifically: The importance weight λ is used to balance the classification loss _LS from the spatial component contribution and the classification loss _LT from the spatial component contribution, as follows: L _total = λ × _LS + (1-λ) × _LT . 8. A system for implementing the video facial expression recognition method based on face key point optimization regional features of any one of the methods of claims 1 to 7, characterized in that the system comprises: The spatial feature extraction module and the feature extraction module based on facial key points are used to extract spatial feature data and facial key point feature data from the relevant data information in the collected video facial expression recognition data set; A face key point guided graph convolution module, connected to the spatial feature extraction module and the face key point based feature extraction module, for enhancing the facial expression information in the frame with the help of face key points in the form of graph convolution; A feature fusion module, connected to the face key point guided graph convolution module, is used to compress high-dimensional features and determine the key feature vectors of emotional information according to face area division; The timing module and the classification module are connected to the feature fusion module, and a timing feature extraction network is constructed using a multi-head self-attention mechanism MHSA and a multi-layer MLP. The classification results are supervised with labels using the classification network and the cross entropy loss function to obtain the final video facial expression recognition results. 9. A device for realizing video facial expression recognition based on face key point optimization regional features, characterized in that the device comprises: a processor configured to execute computer-executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, each step of the video facial expression recognition method based on optimizing regional features of facial key points described in any one of claims 1 to 7 is implemented. 10. A processor for implementing video facial expression recognition based on optimized regional features of facial key points, characterized in that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the various steps of the video facial expression recognition method based on optimized regional features of facial key points described in any one of claims 1 to 7 are implemented. 11. A computer-readable storage medium, characterized in that a computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the video facial expression recognition method based on optimizing regional features of facial key points as described in any one of claims 1 to 7.