CN111881815A - A face detection method based on multi-model feature transfer - Google Patents

Abstract

The present invention proposes a face live detection method with multi-model feature migration. By constructing and fusing heterogeneous data sets, the multi-model feature migration method in multiple color spaces is used for live body training, so as to improve the accuracy and generality of the live body detection model. transformation ability. The specific method is, in the training phase, fuse visible light images from open source or private datasets, and then train the RGB model and YUV model separately after face detection, alignment, and cropping until the models converge; in the prediction phase, the collected visible light images are collected. Input to the trained RGB model and YUV model respectively, obtain the results of the two models respectively, obtain the final score through the model score fusion strategy, and finally judge the living body detection result according to the score. The method has good generalization performance and high accuracy, and is suitable for industrial deployment.

Description

A face detection method based on multi-model feature transfer

technical field

The invention belongs to the technical fields of computer vision, pattern recognition, machine learning, convolutional neural network, and face recognition, and in particular relates to a face living body detection method with multi-model feature migration.

Background technique

Face recognition technology has been widely used in security monitoring, human-machine exchange, e-commerce, mobile payment and other fields, and face liveness detection is the first threshold of face recognition, and it is also the premise of the application of face recognition technology. In the current live detection, the main technical solutions are interactive live detection, multi-source information fusion live detection, and static image live detection. Interactive live detection requires user cooperation, which is very inconvenient, cumbersome steps, easy for users to resist, and low efficiency; multi-source information fusion live detection often requires adding depth cameras, infrared cameras, 3D cameras, microphones, etc., which not only increases hardware overhead At the same time, it also brings a lot of complex 3D modeling calculations; static image live detection is often a low-cost and fast live detection method, but due to the lack of current data sets and inefficient model construction methods, it often leads to model generalization insufficient.

In the current static image liveness detection methods, the liveness detection model is often constructed for a limited number of attack types in a single scene represented by a single data set. This method can achieve ideal accuracy requirements in the laboratory stage. , but the actual scene in industrial and real use is very complex, not only the lighting and background brought by the variety of scenes, but also the variety of attack types and attack methods, which can really be used for live detection. posed serious challenges. In common living attack detection, the printer type, print quality, different display types, resolutions, sizes, etc. brought by different display devices, and even the angle, distance, display brightness, and whether the display device is filmed are all displayed. will have an impact on live detection. The diversity of attack types and the large differences in image distribution in the same attack lead to low generalization ability of the model in real scenarios. Aiming at the lack of generalization ability of the past live detection methods, a multi-model transfer live detection method is proposed. By constructing heterogeneous data sets, the multi-model feature transfer fusion method in multiple color spaces is used for live training and prediction.

Patent CN109840467A uses a so-called generative adversarial network (GAN) to generate a new training set of negative samples (the negative samples are attack images), the purpose is to solve the problem that the number of negative samples is too small when using deep learning to train the network. question. But on a given dataset, the GAN method can only learn a limited probability distribution of samples on the dataset. Therefore, for brand-new attack scenarios and means, the image data generated by this network has limited representation and insufficient generalization ability.

The patent CN110472519A adopts a multi-model fusion method for live detection, but its model requires both natural light images and infrared light images to be input to the network as training sets. In this method, the infrared light image acquisition method is complicated, requires an infrared light camera, increases the hardware cost, and is not conducive to the rapid upgrade of some existing face detection and recognition equipment.

In the existing technology, due to the single distribution of training data and the inefficient model building method, the static live detection model often has the problem of insufficient generalization, and generally cannot be used in industrial and real production scenarios.

SUMMARY OF THE INVENTION

Aiming at the problem of insufficient generalization in the current static live body detection model, the present invention proposes a face live body detection method with multi-model feature migration.

The present invention is achieved through the following technical solutions:

A face liveness detection method with multi-model feature migration, the method comprises the steps:

S1, obtain a visible light image, and the visible light includes a human face;

S2, using the first RGB model to identify the visible light image to obtain the first living body probability;

S3, using the second YUV model to identify the visible light image to obtain a second living body probability;

S4, determining a third living body probability according to the first living body probability and the second living body probability; S5, judging whether it is a living body according to the third living body probability.

Further, the first living body probability includes: a negative sample probability p1 and a positive sample probability p2; the second living body probability includes: a negative sample probability p3 and a positive sample probability p4.

Further, step S4 also includes: the third living body probability is the mean value of each model probability, that is, the final negative sample probability is α×(p1+p3), and the final positive sample probability is β×(p2+p4), where 0≤α , β≤1, α+β=1.

Further, step S4 also includes: judging step by step, setting the threshold of the first RGB model to be T1 and the threshold of the second YUV model to be T2, where 0.5≤T1<1, 0.5≤T2<1; specifically, first determine the first RGB model The output of the model, if p2<1-T1 or p2>T1, then the final output probability of the third living body is the first living body probability output by the first RGB model, if not, then judge the output of the second YUV model; if In the second YUV model, if p4<1-T2 or p4>T2, the final output probability of the third living body is the second living body probability output by the second YUV model; if not, the third living body probability is the probability of each model The mean of , that is, the final negative sample probability is α×(p1+p3), and the final positive sample probability is β×(p2+p4), where 0≤α, β≤1, α+β=1.

Further, step S5 includes: setting the living body judgment threshold as T3, where 0.5≤T3<1, if the probability of the final positive sample is greater than or equal to T3, the image is judged as a positive sample; if it is less than T3, then it is judged as a negative sample .

Further, before step S1, step S0 is also included, in the training phase, in the training phase, the visible light images on the heterogeneous data sets are fused, and the first RGB model and the second YUV model are trained respectively after face detection, alignment and cropping. until the model converges.

Further, in step S0, the training phase includes the following steps:

S101: Construction of heterogeneous data sets, collecting heterogeneous data sets, and selecting only images or videos under visible light to form heterogeneous data sets; positive samples are real samples in heterogeneous data sets, and negative samples are attack samples in heterogeneous data sets ;

S102: Data preprocessing, including 3 steps:

A: Face detection, for video data, perform face detection every n frames, if a face is detected, proceed to the next step, otherwise, perform the next face detection every n frames; for image data, Perform face detection directly, if a face is detected, proceed to the next step, otherwise, perform face detection in the next image;

B: face alignment, which is characterized by adopting similarity transformation to align the faces detected in step A;

C: face cropping, crop the face aligned in step B to an input size suitable for both the first RGB model and the second YUV model;

S103: Train the first RGB model, and input the preprocessed face image of S102 into the first RGB model for training;

S104: Train the second YUV model, convert the preprocessed face image in S102 to the color space, first transfer the face image from the RGB color space to the YUV color space, and then input it into the second YUV model for training;

S105: Train the two models in S103 and S104 respectively. When the model converges and reaches the expected accuracy on the validation set or the test set, it means that the model training is completed, and the process goes to step S1.

Further, in step S101, the heterogeneous data sets include public or private data sets, wherein the public data sets include: Replay-Attack, Print-Attack, Yale-Recaptured, CASIA-MFSD, MSU-MFSD, Replay- Mobile, Msspoof, SiW, Oulu-NPU, VAD, NUAA or CASIA-SURF.

A computer-readable storage medium on which a computer program is stored, wherein, when the program is executed by a processor, the steps of a method for detecting a face living body with multi-model feature migration are realized.

A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a method for facial living detection using multi-model feature migration when the processor executes the program.

The key points of the present invention are:

1. Construction strategies and methods of heterogeneous data sets: The construction strategies of heterogeneous data sets and multi-models in various color spaces are complementary. If there is no heterogeneous data set, multi-models in various color spaces learn. The features are limited and single, and the generalization ability is low;

2. Model construction scheme for multi-model feature transfer in RGB and YUV color spaces: lack of multi-model in multiple color spaces, complex features in heterogeneous datasets are difficult to be fully learned by a single model;

3. Model score fusion strategy: The model score fusion strategy is a key factor affecting the final effect of the multi-model. If the model score fusion strategy does not match the actual multi-model, the final effect of the multi-model is often worse than that of a single model.

Compared with the prior art, the present invention has at least the following beneficial effects or advantages: firstly, the solution has the advantages of low cost and small calculation amount, and can quickly deploy and upgrade some existing single-camera face detection and recognition equipment. This solution adopts static image living detection technology, only needs a single camera, no cost caused by additional hardware such as depth camera, infrared camera, 3D camera, microphone, etc., without a large number of complex 3D modeling calculations, and has the characteristics of low cost and small amount of calculation. . The backbone network used in this solution can be replaced with lightweight networks such as MobileNet V1, MobileNet V2, and EfficientNet according to actual needs, thereby further speeding up inference calculations. Secondly, the live detection model established by this scheme has the advantages of strong generalization and high accuracy. Because the training set uses heterogeneous data sets and builds multiple models in various color spaces, the generalization and accuracy of the model are obtained. significantly improved.

Description of drawings

The present invention will be described in further detail below in conjunction with the accompanying drawings;

Fig. 1 is the flow chart of the training phase of the present invention;

Fig. 2 is the flow chart of the prediction stage of the present invention;

FIG. 3 is a flow chart of the step-by-step judgment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Aiming at the problem of insufficient generalization of existing deep learning-based static live detection models, a live detection method based on multi-model feature transfer was proposed. By constructing and fusing heterogeneous datasets, the multi-model feature transfer fusion method in multiple color spaces is used for in vivo training to improve the accuracy and generalization ability of the in vivo detection model. The specific method is that, in the training phase, the visible light images on the open source or private data sets are fused, and after face detection, alignment, and cropping, the RGB model and the YUV model are trained at the same time; in the prediction phase, the collected visible light images are respectively input into RGB Model and YUV model, respectively obtain the results of the two models, obtain the final score through the threshold fusion scheme, and finally judge the living body detection result according to the score. The method has good generalization performance and high accuracy, and is suitable for industrial deployment.

In one embodiment of the present invention, there is provided a face liveness detection method with multi-model feature migration, comprising the steps of:

S1, obtain a visible light image, and the visible light includes a human face;

S2, using the first RGB model to identify the visible light image to obtain the first living body probability;

S3, using the second YUV model to identify the visible light image to obtain a second living body probability;

S4, determining a third living body probability according to the first living body probability and the second living body probability; S5, judging whether it is a living body according to the third living body probability.

Further, the first living body probability includes: a negative sample probability p1 and a positive sample probability p2; the second living body probability includes: a negative sample probability p3 and a positive sample probability p4.

Further, step S4 also includes: the third living body probability is the mean value of each model probability, that is, the final negative sample probability is α×(p1+p3), and the final positive sample probability is β×(p2+p4), where 0≤α , β≤1, α+β=1.

Further, step S4 also includes: judging step by step, setting the threshold of the first RGB model to be T1 and the threshold of the second YUV model to be T2, where 0.5≤T1<1, 0.5≤T2<1; specifically, first determine the first RGB model The output of the model, if p2<1-T1 or p2>T1, then the final output probability of the third living body is the first living body probability output by the first RGB model, if not, then judge the output of the second YUV model; if In the second YUV model, if p4<1-T2 or p4>T2, the final output probability of the third living body is the second living body probability output by the second YUV model; if not, the third living body probability is the probability of each model The mean of , that is, the final negative sample probability is α×(p1+p3), and the final positive sample probability is β×(p2+p4), where 0≤α, β≤1, α+β=1.

Further, step S5 includes: setting the living body judgment threshold as T3, where 0.5≤T3<1, if the probability of the final positive sample is greater than or equal to T3, the image is judged as a positive sample; if it is less than T3, then it is judged as a negative sample .

Further, before step S1, step S0 is also included, in the training phase, in the training phase, the visible light images on the heterogeneous data sets are fused, and the first RGB model and the second YUV model are trained respectively after face detection, alignment and cropping. until the model converges.

In another embodiment, the method is divided into two steps, the first step is the construction of the living body detection model, which corresponds to the training stage; the second step is the deployment of the living body detection model, which corresponds to the prediction stage. The scheme of the training phase is shown in Figure 1, and the scheme of the prediction phase is shown in Figure 2.

The training phase is characterized by constructing heterogeneous datasets and simultaneously training two or more deep models with different architectures, each deep learning model using a different color space.

S101: Construction of heterogeneous datasets. First, public or private datasets are collected, and only images or videos under visible light are selected to constitute heterogeneous datasets. Public datasets include, but are not limited to, the following datasets: Replay-Attack, Print-Attack, Yale-Recaptured, CASIA-MFSD, MSU-MFSD, Replay-Mobile, Msspoof, SiW, Oulu-NPU, VAD, NUAA, CASIA -SURF et al. For private specific scenarios, collect and join private datasets. The positive samples are real samples in the heterogeneous data set, the negative samples are attack samples in the heterogeneous data sets, and the negative sample types refer to print image attacks, tablet attacks, mobile phone attacks, 2D attack types such as display attacks. Construction strategies for heterogeneous datasets include the following two:

Let the collected data sets be S, denoted as {D ₁ , D ₂ , D ₃ ,..., D _S }. Let the number of positive samples contained in the mth data set D _m be M _m , the number of negative samples be N _m , and there are O _m types of negative samples, where m={m|1≤m≤S, m∈N ^* }. Let γ be the equalization factor, where 0≤γ≤1.

个样本，那么负样本总共抽取γN_m个。如果γN_m＞M_m，正样本随机(分层)收取M_m个；如果γN_m≤M_m，正样本随机(分层)抽取γN_m个。Strategy 1: In the mth dataset, a negative sample type is randomly (stratified) drawn

samples, then a total of γN _m negative samples are drawn. If γN _m >M _m , M _m positive samples are randomly (stratified) collected; if γN _m ≤M _m , γN _m positive samples are randomly (stratified) selected.

正样本随机(分层)抽取

个；如果

正样本随机(分层)抽取γP_oO_o个。Strategy 2: First, the positive samples in all data sets are classified into one category, and the negative samples are classified according to the type of negative samples. It is assumed that there are O _o classes of negative sample types, and each class has P _o negative samples. For example, if the print image attack appears in the {D ₁ , D ₂ , D ₅ } dataset at the same time, then the print image attack in the negative samples of the three datasets {D ₁ , D ₂ , D ₅ } is classified into one category. The specific construction method of strategy 2 is to randomly (hierarchically) extract γP _o samples for each type of negative sample. Then on all the data, the total number of negative samples drawn is γP _o O _o . if

Random (stratified) selection of positive samples

a; if

Positive samples are randomly (stratified) drawn γP _o O _o .

S102: data preprocessing. Data preprocessing consists of 3 steps. The first step is face detection. For video data, face detection is performed every n (n is greater than 1) frames. If a face is detected, the second step is performed. Otherwise, the next n frames are performed. Face detection; for image data, face detection is performed directly. If a face is detected, the second step is performed, otherwise, the face detection of the next image is performed. The second step is face alignment, which is characterized by using similarity transformation to align the faces detected in the first step. The third step is face cropping, and the face aligned in the second step is cropped to an input size suitable for both the RGB model and the YUV model.

S103: Train an RGB model. The preprocessed image of S102 is input into the RGB model for training. The RGB model refers to a deep learning model whose color space of the input image is RGB. The specific network backbone of the RGB model can be, but is not limited to, VGG, GoogLeNet, ResNet, DenseNet, MobileNet V1, MobileNet V2, MobileFaceNet, EfficientNet, ShuffleNet and other volumes Integrating Neural Networks and Their Variants. In particular, the RGB model here is pretrained on the ImageNet dataset.

S104: Train a YUV model. The preprocessed image of S102 is converted into color space, and the image is first transferred from RGB color space to YUV color space, and then input into the YUV model for training. The YUV model refers to a deep learning model in which the color space of the input image is YUV. The YUV color space specifically includes, but is not limited to, YCrCb, YCbCr, YPbPr, YDbDr, and the like. The specific network backbone of the YUV model can be, but is not limited to, convolutional neural networks such as VGG, GoogLeNet, ResNet, DenseNet, MobileNet V1, MobileNet V2, MobileFaceNet, EfficientNet, ShuffleNet and their variants. In particular, the YUV color space here refers to YCrCb, and the YUV model is pre-trained on the ImageNet dataset.

S105: Model training is completed. The two models in S103 and S104 are trained respectively. When the model converges and reaches the expected accuracy on the validation set or test set, it means that the model training is completed, and the deployment of the next prediction stage can be performed.

In the prediction stage, after the input visible light image is preprocessed, it is input to the RGB model and the YUV model after color space conversion, and each obtains a score, and then the final score is obtained through the score fusion scheme, so as to judge the final live detection. model results. The specific steps of the inference stage are:

S201: Input a visible light image. The RGB image containing the human face is collected by the visible light camera, and the image is the input of S202.

S202: Image preprocessing. The RGB images collected in S201 are preprocessed, and the preprocessing method is the same as that in S102.

S203: RGB model forward calculation. The image preprocessed in S202 is sent to the trained RGB model for forward calculation.

S204: Obtain the RGB model score. Obtain the network output result after forward calculation in S203, set the output probability of negative sample as p1 and the probability of positive sample as p2, denoted as (p1, p2), where 0≤p1≤1, 0≤p2≤1, p1+p2 =1.

S205: Color space conversion. The image preprocessed by S202 is converted into YUV color space after color space conversion. In particular, the YUV color space here refers to YCrCb.

S206: YUV model forward calculation. Send the image converted by the color space of S205 to the already trained YUV model for forward calculation.

S207: Obtain the YUV model score. Obtain the network output result after forward calculation in S206, set the output probability of negative samples as p3 and the probability of positive samples as p4, denoted as (p3, p4), where 0≤p3≤1, 0≤p4≤1, p3+p4 =1.

S208: Model score fusion. Model score fusion strategies include the following two:

Strategy 1: The final probability output is the mean of the probabilities of each model. Specifically, the final probability of negative samples is α×(p1+p3), and the probability of positive samples is β×(p2+p4), denoted as (α×(p1+p3),β×(p2+p4)), where 0 ≤α, β≤1, α+β=1. Typically, α=0.5, β=0.5.

Strategy 2: Judgment step by step. The process is shown in Figure 3. The RGB model threshold is set to T1 (0.5≤T1<1), and the YUV model threshold is T2 (0.5≤T2<1). The specific process is to first judge the output of the RGB model. If p2<1-T1 or p2>T1, the final output is the output of the RGB model; if not, the output of the YUV model is judged. If in the YUV model, if p4<1-T2 or p4>T2, the final output is the output of the YUV model; if not, the final model output is equivalent to the probability output of the above strategy 1, that is, the final output probability of negative samples is α×(p1+p3), the positive sample probability is β×(p2+p4), denoted as (α×(p1+p3),β×(p2+p4)), where 0≤α,β≤1,α +β=1. Typically, α=0.5, β=0.5.

S209: Judging the result of the living body detection model. Set the living body judgment threshold as T3 (0.5≤T3<1). If the probability of the positive sample in the output of S208 is greater than or equal to T3, the image is judged as a living body image (positive sample); if it is less than T3, it is judged as an attack image (negative sample). sample). For the model thresholds T1, T2 and T3 described above, typically, T1=T2=T3.

The present invention also provides a computer-readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, the steps of the method for detecting a face living body with multi-model feature migration are realized.

The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements multi-model feature migration for face liveness detection when the processor executes the program steps of the method.

The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above-mentioned specific embodiments are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention. protected range. Any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention also fall within the protection scope of the present invention.

Claims (10)

1. A human face living body detection method of multi-model feature migration is characterized by comprising the following steps:

s1, acquiring a visible light image, wherein the visible light comprises a human face;

s2, identifying the visible light image by using a first RGB model to obtain a first living body probability;

s3, identifying the visible light image by using a second YUV model to obtain a second living body probability;

s4, determining a third living body probability according to the first living body probability and the second living body probability;

and S5, judging whether the living body is the living body according to the third living body probability.

2. The method of claim 1, wherein the first live probability comprises: negative sample probability p1 and positive sample probability p 2; the second live body probability includes: negative sample probability p3 and positive sample probability p 4.

3. The method according to claim 2, wherein step S4 further comprises: the third live body probability is the mean of the probabilities of the respective models, i.e., the final negative sample probability is α × (p1+ p3) and the final positive sample probability is β × (p2+ p4), where 0 ≦ α, β ≦ 1, and α + β ≦ 1.

4. The method according to claim 2, wherein step S4 further comprises: step-by-step judgment, setting a first RGB model threshold value as T1 and a second YUV model threshold value as T2, wherein T1 is more than or equal to 0.5 and less than 1, and T2 is more than or equal to 0.5 and less than 1; specifically, the output of the first RGB model is firstly determined, if p2 is less than 1-T1 or p2 is greater than T1, the finally output third live body probability is the first live body probability output by the first RGB model, and if not, the output of the second YUV model is determined; if p4 < 1-T2 or p4 > T2 in the second YUV model, the final output third living body probability is the second living body probability output by the second YUV model, and if not, the third living body probability is the average value of the model probabilities, namely the final negative sample probability is alpha x (p1+ p3) and the final positive sample probability is beta x (p2+ p4), wherein alpha is greater than or equal to 0, beta is less than or equal to 1, and alpha + beta is 1.

5. The method according to claim 1, wherein step S5 includes: setting the living body judgment threshold value as T3, wherein T3 is more than or equal to 0.5 and less than 1, and if the probability of the final positive sample is more than or equal to T3, judging the image as the positive sample; if the value is less than T3, the result is judged to be a negative sample.

6. The method of claim 1, further comprising a step S0 before the step S1, wherein the training phase is a step of fusing visible light images on the heterogeneous data set, and performing face detection, alignment and cropping, and training the first RGB model and the second YUV model respectively until the models converge.

7. The method of claim 6, wherein in step S0, the training phase comprises the steps of:

s101: constructing a heterogeneous data set, collecting the heterogeneous data set, and only selecting images or videos under visible light to form the heterogeneous data set; the positive sample is a real sample in the heterogeneous data set, and the negative sample is an attack sample in the heterogeneous data set;

s102: data preprocessing, comprising 3 steps:

a: face detection, namely performing face detection once every n frames for video data, if a face is detected, performing the next step, and if not, performing the face detection of the next n frames; directly carrying out face detection on image data, if a face is detected, carrying out the next step, and otherwise, carrying out face detection on the next image;

b: b, face alignment, which is characterized in that the face detected in the step A is aligned by adopting similarity transformation;

c: b, cutting the face aligned in the step B to an input size suitable for both the first RGB model and the second YUV model;

s103: training a first RGB model, and inputting the preprocessed face image of S102 into the first RGB model for training;

s104: training a second YUV model, converting the preprocessed face image of S102 from an RGB color space to a YUV color space through color space conversion, and inputting the face image into the second YUV model for training;

s105: and respectively training the two models in the S103 and the S104, and when the models converge and reach the expected accuracy on the verification set or the test set, representing that the model training is finished, and entering the step S1.

8. The method according to claim 7, wherein in step S101, the heterogeneous data set comprises a public or private data set, wherein the public data set comprises: Replay-Attack, Print-Attack, Yale-Recampled, CASIA-MFSD, MSU-MFSD, Replay-Mobile, Mspoof, SiW, Ouu-NPU, VAD, NUAA, or CASIA-SURF.

9. A computer-readable storage medium, having stored thereon a computer program, wherein the program, when executed by a processor, performs the steps of the method for living human face detection with multi-model feature migration according to any one of claims 1-8.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for living human face detection with multi-model feature migration according to any one of claims 1-8.

Images