CN100361138C - Method and system for real-time detection and continuous tracking of human faces in video sequences - Google Patents

Abstract

The present invention proposes a method and system for real-time detection and continuous tracking of human faces in video sequences. The two-level face detection algorithm detects and verifies the detected faces; uses the object tracking algorithm to track the verified faces; and verifies the tracked faces through the verification of the tracking area. The present invention adopts the human face detection method based on the AdaBoost statistical layered classifier to realize the real-time detection of the frontal upright human face, and combines the human face tracking method based on Mean shift and histogram features to realize the real-time detection and tracking of the human face, The invention has the advantages of rapid detection and tracking and strong real-time performance.

Description

Method and system for real-time detection and continuous tracking of human faces in video sequences

technical field

The invention relates to the field of human face detection and tracking, in particular to a method and system for real-time detection and continuous tracking of human faces in video sequences.

Background technique

Human face is one of the most convenient ways of human-computer interaction in computer vision system. Face detection is to determine the position, size and other information of all faces in an image or image sequence, while face tracking is to continuously track one or more detected faces in a video sequence. Face detection and tracking technology is not only a necessary prerequisite for face recognition, expression recognition, face synthesis and other technologies, but also has a wide range of application values in intelligent human-computer interaction, video conferencing, intelligent monitoring, video retrieval and other fields.

The importance of face detection and the complexity of face patterns have made face detection a research hotspot in the field of computer vision. There are many related literatures and methods, which are mainly divided into two categories based on heuristic rules and based on statistical models. The method based on heuristic rules first extracts the features with clear physical meaning such as the edge of the face, skin color, movement, symmetry, contour, facial organs, etc., then formulates a series of rules according to the prior knowledge of the face, and finally passes the test. Whether the features conform to these prior rules is used to detect faces. This type of method is generally faster, but it depends on fixed prior rules, poor ability to adapt to changes, and more false alarms. The statistical method uses a large number of "face" and "non-face" samples, trains and constructs a classifier through the pixel grayscale features or other transformation domain features, and uses the constructed classifier to classify all Candidate face regions of possible sizes are judged to detect faces of all possible positions and sizes.

Although the above method is trained with a large number of samples, it is more reliable in the statistical sense, expands the detection range, improves the robustness of the detection system, and is more suitable for face detection in complex scenes, but the time required is relatively large. Long, real-time performance is relatively poor.

Although the face detection algorithm of AdaBoost is also used in the prior art, it can be seen based on the above method that the training time is too long and the number of extracted face features is too large. The position of the frame target only considers the histogram of the previous frame, and uses the histogram of the previous frame as a template, which will easily lead to some unstable results. If the tracking result of a certain frame is not accurate enough, the tracking results of subsequent frames will also continue. If there is a mistake, there is also the problem of poor ability to detect faces with uneven illumination on the left and right, and is easily disturbed by noise, and the stability is relatively poor.

And because the image is a video sequence input by the video camera, this means that the face input to the camera has great uncertainty, and is often disturbed by expressions and appearance. Beards, glasses, and hair can also affect the appearance of the face. In addition, changes in face size, rotation, attitude, pitch, occlusion in local areas, and changes caused by imaging conditions all greatly affect the appearance of the face, thereby affecting the performance of the algorithm.

Contents of the invention

The present invention solves the technical problems of the method for real-time detection and continuous tracking of human faces in the video sequences collected in the prior art and the system's relatively long calculation time and poor real-time performance. The face detection method of the classifier realizes the real-time detection of the upright face of the front, and in combination with the face tracking method based on Mean shift and histogram features, realizes the real-time detection and tracking of the face. The method of the present invention has the advantages of detection and tracking The advantages of rapidity and strong real-time performance.

The purpose of the present invention is achieved like this:

A method for real-time detection and continuous tracking of human faces in a video sequence, comprising the following steps:

input video image;

Use the real-time face detection algorithm to detect the face of the input video image;

Then use the thick and thin two-level face detection algorithm to detect and verify the detected faces;

Use the object tracking algorithm to track the verified face;

The tracked face is verified by verifying the tracked area.

The real-time face detection algorithm is realized by a multi-level classifier based on the AdaBoost algorithm.

Described face detection comprises the following steps:

The received image information is zoomed in and out, and the face window is searched;

Perform feature point positioning on the detected face, and perform geometric normalization on the face;

Perform gray level equalization processing on the face;

Rotate and scale the face;

Get the detected standard face image.

The step of detecting and verifying the detected faces by using the thick and thin two-level face detection algorithm includes the following situations:

If one or more faces are detected in a frame of images, track these faces in the next two frames of images, and detect and verify the faces tracked in the next two frames of images;

After a face is detected in three consecutive frames at a certain position, it is determined that the face exists, and one of them is selected to start tracking.

The method of the present invention further includes the step of normalizing the gray levels on both sides of the face after performing gray level equalization processing on the face, so that the mean value and variance of the gray levels on the left and right sides of the face are equal.

After the face rotation and scaling, it also includes the steps of micro-feature calculation and classifier judgment. This step is to use the integral map and square integral map of the processed face image to obtain microstructural features of any scale and position in the image.

The described steps of verifying the detected faces include:

Continue to track the selected face in the subsequent frames. If the similarity between the tracking results of the next frame and the previous frame in the adjacent frames is lower than the set value, stop tracking; after the previous target stops tracking, start again in the subsequent images. Face detection until a new face is found, followed by a tracking step after verification.

In the step of using the thick and thin two-stage face detection algorithm to detect and verify the detected faces, the face detection is realized in two stages, and the resolution of the face window of the coarse detection search is smaller than that of the fine detection search. The resolution of the face window, for each scale of the image, first reduce it accordingly, use the face window searched by the coarse detection to detect the face, eliminate the non-face window, and then in the original scale image, use the fine detection The searched face window performs face detection on the remaining face candidate windows.

In the step of face tracking after verification, the object tracking algorithm is based on Meanshift and histogram features.

The histograms include long-term histograms, short-term histograms and color histograms.

The step of verifying the tracking area refers to performing face detection in the tracking area every few frames. If a frontal face is detected in the tracking area, the tracking parameters are updated according to the size and position of the detected face. The tracking parameters include The center, radius and histogram features of the target; if the target is tracked for hundreds of consecutive frames, but no frontal face is detected in the tracking area, stop tracking, and perform face detection in subsequent images until a new face is found , followed by a trace step after validation.

The present invention also proposes a real-time detection and continuous tracking system for human faces in video sequences, including a human face detection device and a human face tracking device, and the human face detection device includes a human face processing unit, a micro-feature calculation unit and a classifier unit; The face processing unit described above receives the image information, zooms the received image, exhaustively searches the candidate face window, and calculates the mean value and variance of the window gray scale; Microstructural features, and send it to the described classifier unit for judgment, the classifier unit adopts thick and thin two-level face detection algorithm to make a judgment and then send it to the human face tracking device; the described human face tracking device includes object The tracking unit and the tracking area verification unit, the object tracking unit uses the histogram feature for calculation to realize image tracking, the tracking area verification unit performs area detection on the tracked image, and verifies the tracked face.

The classifier unit of the described face detection device also includes a thick and thin two-level face detection unit, which receives microstructure features and performs thick and thin two-level detection: the resolution of the human face window searched by the coarse detection is smaller than that of the human face searched by the fine detection. For the resolution of the face window, for each scale image, first reduce it accordingly, use the face window searched by coarse detection for face detection, eliminate non-face windows, and then use fine detection search in the original scale image Face detection is performed on the remaining face candidate windows to determine the face window.

The histograms used by the object tracking unit include long-term histograms, short-term histograms and color histograms.

The technical effect that the present invention produces is remarkable:

The method and system of the present invention are based on the AdaBoost algorithm for detection, and the coarse model and the fine model are used for face detection and verification, so that the present invention can achieve a very high detection rate, and the detection speed is very fast, the efficiency is high, and real-time Realize; the present invention carries out the normalization of grayscale to left and right people's face resolution according to the mean value and the variance of standard people's face grayscale in detection process, eliminates the influence of uneven illumination of left and right people's faces; The present invention is after people's face detection finishes Continue to track and verify faces in a short period of time to eliminate the influence of false alarms in face detection; introduce two local features of long-term histogram and short-term histogram in the process of face tracking to reflect the change process of the target histogram in the previous image , to ensure that the algorithm can track the moving target whose attitude changes constantly.

Description of drawings

FIG. 1 is a flow chart of the face detection and tracking method of the present invention.

2a and 2b are schematic diagrams of the detection and tracking results of the face detection and tracking method of the present invention.

Figure 3 is a schematic diagram of the face detection process based on the AdaBoost hierarchical classifier.

Figure 4 shows some positive sample face images in the classifier.

Figure 5 is a part of the negative sample images that do not contain faces in the classifier.

6a to 6d are schematic diagrams of calibration and collection of face samples.

Figure 7 is a sample after gray normalization processing on both sides.

Figure 8 shows the original sample and the sample after mirroring, left and right rotation, and zooming in.

Figure 9 shows the scale normalized negative sample data.

Fig. 10 shows seven groups of micro-features selected by the embodiment of the face detection algorithm of the present invention.

Fig. 11 is a schematic diagram of the training process of the K-th layer classifier of the AdaBoost training algorithm.

12a and 12b are schematic diagrams of an embodiment of processing results after face detection.

Fig. 13a and Fig. 13b are schematic diagrams of face tracking in the present invention.

FIG. 14 is a schematic diagram of face detection results.

Fig. 15 is a schematic diagram of face detection and tracking results.

Fig. 16 is a block diagram of the system of the present invention.

Fig. 17 is another block diagram of the system of the present invention.

Detailed ways

The present invention proposes a method for real-time detection and continuous tracking of faces in video sequences, combined with the content shown in Figure 1:

At first, input video image, in this step, input video image by camera in real time;

Then, the real-time face detection algorithm is used to detect the face of the input video image;

After receiving the real-time input video image, search each frame of image to detect the existence of human face; as shown in Figure 2a, the result of human face detection is given, where the square box represents the detected human face; in the detection In the process, if one or more faces are detected in a frame of images, these faces will be tracked in the next two frames of images, and the faces tracked in the next two frames of images will be detected and verified, and the previous Whether the detection result is a real face;

Only when a human face is detected in a certain position for three consecutive frames, the algorithm considers that the human face exists at this position. If there are many human faces in the scene, a human face is selected to start tracking; face description;

Continue to track the selected faces in the subsequent frames, if the similarity between the tracking results of the next frame and the previous frame in the adjacent frames is lower than the set value (the set value can be set arbitrarily), then stop tracking; If no frontal upright face is detected in the area where a tracking target is located for a long time, it is considered that the tracking value of the target is not great, and the tracking of the target is stopped; after the previous tracking target stops tracking, face detection is performed again in subsequent images, Until a new face is found, track the new face. Figure 2b shows the results of face tracking.

The following describes the detection process of the present invention in conjunction with the training process of the samples. The present invention uses the statistical training method to detect the front face in the scene, and uses the AdaBoost theory to realize the training of the statistical model of human face detection. The face detection algorithm based on AdaBoost first trains a "face/non-face" classifier from a large number of "face" and "non-face" samples. Whether the rectangular window is a human face or not, let the length of the rectangle be m and the width be n, then the process of face detection is: firstly, the image is continuously zoomed in according to a certain ratio, and all the obtained series of images are exhaustively searched and identified with a size of m× n-pixel window, input each window into the "face/non-face" classifier, leave the candidate window recognized as a face, and then use the candidate of the adjacent position to calculate the combined average value, and output all detections The location, size and other information of the received face.

A kind of real-time human face detection algorithm described in the present invention is to use a kind of microstructural feature similar to Harr wavelet to express human face mode, and proposes a kind of feature selection method in conjunction with above-mentioned AdaBoost algorithm, multiple based on single feature The weak classifiers are composed into a strong classifier, and then multiple strong classifiers are cascaded into a complete face detection classifier, as shown in Figure 3. Each layer of classifiers formed in the detection process in the present invention is a strong classifier trained by the AdaBoost algorithm, and each layer of strong classifiers is composed of a certain number of weak classifiers. When detecting, if a strong classifier of a certain layer judges a sub-window as False, the sub-window will be excluded without further discrimination. If the output is True, the next layer of more complex classifiers will be used to discriminate the sub-window. During the search process of candidate windows, each layer of strong classifiers can pass almost all face samples and reject most non-face samples. In this way, there are more windows input to the low-level strong classifier, while the windows input to the high-level are greatly reduced.

The face detection algorithm needs to scale the image after the exhaustive search to a certain scale to obtain the face window. In this embodiment, take the image of 320×240 pixels as an example, such as reducing it 10 times according to the ratio of 1.25, and using 20 A window of size ×20 is searched pixel by pixel, and a total of more than 170,000 windows need to be judged. This means that a face detection needs to search a large number of windows. Faced with such a large amount of calculation, the low-level classifiers in the detection process in Figure 3 must be very simple, that is, the number of weak classifiers contained in the previous classifiers is small. The latter classifier is complex and contains a large number of weak classifiers, so that the latter layer can use more features to reject candidate windows similar to human faces, thereby ensuring a lower false detection rate.

When performing face detection, the detected image is compared with the training sample data. When performing model training, a large number of face positive samples and negative sample data need to be collected. Through the collection of these data, the method of the present invention is established. Sample database, the positive samples in this database contain multiple face samples, these samples include faces with different expressions, different skin colors, and different ages, including faces with depth rotations from -20° to 20°, including those with and without glasses Some samples of human faces are shown in Figure 4. The negative sample data is a large number of images that do not contain faces, including landscape images, animals, text, etc., see the content shown in Figure 5. During training, analyze the key feature points of all faces, and determine the center of the two eyes, the tip of the nose, the center of the mouth, and the chin of each positive sample face. Geometrically normalize each face according to these calibration points, that is, correct the main organ position of the face image to the standard position, reduce the scale, translation and plane rotation differences between samples, and then cut out the face area according to the organ position as Face samples, so that the face samples introduce as little background interference as possible, and the organ positions of different face samples are consistent. In the process of face detection, it is also necessary to determine the position of the organs of the detected face for geometric normalization of the face.

Referring to the contents shown in Figures 6a to 6d, it shows a standard face image to perform geometric normalization on each face sample and the process of clipping the face area: first, determine that the detection window scale m×n provided earlier is 20 ×20, and then obtain a standard frontal face image. In the standard image, the y coordinates of the two eyes are consistent, and the face is completely symmetrical, as shown in Figure 6a, and the five key feature points of the image are calibrated. The position of the cropped square face area is determined according to the distance and position of the two eyes in the image. Suppose the distance between the two eyes is r, the center point of the connecting line between the two eyes is (x _center , y _center ), and the length and width of the acquisition rectangle are set to 2r, which is twice the distance between the eyes, then the coordinates of the rectangular clipping area (x _left , y center _top , x _right , y _bottom ) are:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; left</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; top</span>}}\\ {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; right</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; bottom</span>}}\end{array}\right\}<span\; class="notranslate">\; =</span>\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; center</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; r</span>}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; top</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 0.5</span>}\mathrm{<span\; class="notranslate">\; r</span>}\\ {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; center</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; r</span>}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; bottom</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1.5</span>}\mathrm{<span\; class="notranslate">\; r</span>}\end{array}\right\}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Normalize the cropped face area to a size of 20×20, as shown in Figure 6b, and obtain the coordinates [x _stad (i), y _stad (i)] of five calibration points after normalization, i=0, 1, 2, 3, 4.

Any given original face sample and five marked feature points [x _label (i), y _label (i)], i=0, 1, 2, 3, 4, as shown in Figure 6c, calculate these five points and The affine transformation coefficient between the five coordinates of the standard image after normalization. During the transformation process, it should be ensured that the overall shape of each face sample remains unchanged, that is, long faces should still be long faces, and short faces should still be short faces, so that the detection algorithm can detect different types of faces, so the affine transformation formula cannot Adding the stretch transformation in all directions of the face, we only consider the two transformations of rotation and overall scaling, so we can determine the affine transformation formula as:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; label</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; label</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\end{array}\right\}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; cos</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}\\ \mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}& \mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; cos</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}\end{array}\right\}\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; stad</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; stad</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\end{array}\right\}<span\; class="notranslate">\; +</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; a</span>}\\ \mathrm{<span\; class="notranslate">\; b</span>}\end{array}\right\}<span\; class="notranslate">\; =</span>\left\{\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; c</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}& \mathrm{<span\; class="notranslate">\; a</span>}\\ \mathrm{<span\; class="notranslate">\; d</span>}& \mathrm{<span\; class="notranslate">\; c</span>}& \mathrm{<span\; class="notranslate">\; b</span>}\end{array}\right\}\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; stad</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; stad</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right\}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0,1,2,3,4</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

The transformation parameters (a, b, c, d) in the above formula can be obtained by using the least square method.

Let the cropped face image be I, and the size of the image is 20×20, then the coordinates (x _ori , y _ori ) of any point (x, y) in the image to the corresponding point in the original sample can be calculated by the affine transformation coefficient .

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; ori</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ori</span>}}\end{array}\right\}<span\; class="notranslate">\; =</span>\left\{\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; a</span>}& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}& \mathrm{<span\; class="notranslate">\; a</span>}\\ \mathrm{<span\; class="notranslate">\; d</span>}& \mathrm{<span\; class="notranslate">\; c</span>}& \mathrm{<span\; class="notranslate">\; b</span>}\end{array}\right\}\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; x</span>}\\ \mathrm{<span\; class="notranslate">\; the\; y</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right\}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

In order to eliminate the influence of noise, the pixel value of (x, y) in the cropped image is set to the mean value of the pixel values of all points within the neighborhood of the corresponding point (x _ori , y _ori ). Thus, the pixel values of all points in I can be obtained, as shown in Figure 6d.

In the process of detection, due to factors such as external lighting and imaging equipment, the brightness or contrast of the face image may be abnormal, and strong shadows or reflections may appear. In addition, there are differences in the skin color of different races. The normalized face samples are subjected to gray level equalization processing to improve their gray level distribution and enhance the consistency between modes. Since in the process of face detection, each search window needs to perform gray level equalization processing, it is not possible to use a computationally complex method for gray level normalization. The invention adopts the image gray mean value and variance normalization which can be realized by a fast algorithm to carry out the gray level equalization of the sample. In addition, due to the influence of different directions, especially the side lighting, there are often obvious differences in the brightness of the left and right faces in the actual scene. Therefore, the present invention normalizes the gray scales of the faces on both sides, so that the mean value and variance of the gray scales of the left and right halves of the face are both equal to a set standard value. Figure 7 shows some face images after normalization processing on both sides.

In order to enhance the robustness of the classifier to detect face rotation and size changes at a certain angle, the present invention performs mirror transformation on each sample, rotates at an angle of ±20°, and enlarges the size by 1.1 times, as shown in Figure 8. Each sample is expanded to five samples, thus forming the positive sample set for AdaBoost training.

The counter-sample data is randomly selected from the counter-sample image library during the training process of each layer of AdaBoost. First randomly determine the serial number of the counter-sample image, randomly determine the size and position of the counter-sample, then cut out the corresponding area in the image corresponding to the serial number, and normalize the cropped image to a size of 20×20 to obtain a counter-sample, as shown in Fig. 9 is part of the counter-sample data.

The present invention adopts a feature extraction method when performing feature calculation. In this embodiment, seven groups of micro-features are used to effectively express the structural characteristics of the human face pattern. Figure 10 shows all the micro-feature structures in the 20×20 image, and the feature is obtained by calculating the difference between the pixel gray value corresponding to the black area and the white area in the image. The size of the black rectangle and the white rectangle in the first six groups of microfeatures remain the same, while the length and width of the white rectangle in the seventh group of microfeatures are three times that of the black rectangle. The length and width of the black rectangle or the white rectangle in each group of micro-features can be selected arbitrarily, that is, the size of each rectangle can be any value from 1 to 20. The position of the central point in each group of micro-features can also be selected arbitrarily, so theoretically 20×20×20×20×7=1120000 features can be obtained in a 20×20 image. Considering that the black or white area in many groups of features has reached outside the 20×20 image, we ignore such features. So the effective number of features is 89199.

In the process of face detection, it is necessary to continuously calculate the micro-features, and input the micro-features into each layer of AdaBoost strong classifier for judgment. Therefore, the micro-feature calculation efficiency determines the efficiency of the face detection algorithm. A microstructural feature of any scale and any position in the image can be quickly obtained by using the integral map and square integral map of the entire image, which provides the possibility for the realization of a real-time face detection system, and this method does not require All pixel values in the 20×20 window are normalized to grayscale, and only the grayscale mean and variance of the left and right half of the 20×20 window are combined to normalize the microstructural features.

As shown in Figure 11, it is the training process of the K-th layer classifier in the AdaBoost algorithm training process. First, all the normalized positive sample face data are input into the classifiers that have been trained in the first K-1 layer, and the positive samples that pass these classifiers are input into the training module of the K-th layer classifier. Then randomly select the anti-sample from the 5,400 anti-sample images mentioned above, and input each anti-sample to the first K-1 layer classifier, and use the samples that passed the K-1 layer classifier as the K-th layer classifier The input counter-sample of the training module of . During the training process, we will ensure that each classifier will not eliminate or eliminate very few positive sample faces. Therefore, the final positive samples input into the training module are basically kept at a certain number. In order to obtain better classification performance and keep the training efficiency from being too low, the present invention selects negative samples equal to the number of positive samples. Therefore, when the number of randomly selected negative samples reaches the equivalent number of positive samples, the selection of negative samples is stopped, and the training of the K-th layer classifier is started.

AdaBoost's strong classifier consists of weak classifiers based on a single feature, that is, each weak classifier corresponds to a micro-feature. The weak classifier of the present invention is defined as:

Where x is a 20×20 image window, g _j (x) represents the feature value of the image under the j feature, θ _j is the decision threshold corresponding to the j feature, h _j (x) represents the image under the j feature The decision output under the feature. The above formula defines a micro-feature j and a threshold θ _j for each weak classifier. There are three possible judgment methods, that is, according to whether the input micro-feature g _j (x) is greater than the threshold, or less than the threshold, or the absolute value is less than This threshold determines whether the decision output is 1 or 0. Each weak classifier can only choose one decision possibility. During the training process, the current micro-features can be processed according to all the positive and negative samples, and the thresholds corresponding to the three kinds of decisions and the classification error rate on the training sample set can be obtained respectively. The method with the smallest error rate is used as the judgment method of the current weak classifier.

Then, each layer of strong classifiers is trained, and the training process is actually the process of obtaining the micro-feature serial numbers corresponding to each weak classifier, so that the overall classification ability of each weak classifier on the training set is the strongest. We set the weak classifiers contained in each classifier to be fixed, set to T, and a micro-feature can only appear once. The micro-features corresponding to the first classifier will not be considered in the subsequent training process. The training process of the algorithm is listed below:

Given a training set {(x _i , y _i )} containing n samples, i=0, 1, ..., n-1{()}, y _i =0 or 1, representing the corresponding input sample x _i is a face sample or a non-face sample, where the number of face samples is m, and the number of non-face samples is 1;

选择误分类风险倍数c，表示训练样本分类错误的风险大小，则对于人脸样本 $C_i=\frac{2c}{m\&CenterDot;(c+1)},$ 对非人脸样本 $C_i=\frac{2}{l\&CenterDot;(c+1)},$ 初始每个样本的权重 $W_0\left(i\right)=\frac{C_i}{\underset{j}{\mathrm{\&Sigma;}}{C}_j},$ c越大，则表示正样本分类错误的风险越大，在训练分类器就应该尽量保证正样本的分类错误率尽可能的小；

Select the misclassification risk multiple c to indicate the risk of misclassification of training samples, then for face samples $C_i=\frac{2c}{m\&Center\; Dot;(c+1)},$ For non-face samples $C_i=\frac{2}{l\&Center\; Dot;(c+1)},$ The initial weight of each sample $W_0\left(i\right)=\frac{C_i}{\underset{j}{\mathrm{\&Sigma;}}{C}_j},$ The larger c, the greater the risk of misclassification of positive samples. When training the classifier, it should try to ensure that the classification error rate of positive samples is as small as possible;

迭代次数t＝0，1，....T-1：(T即为希望选择的弱分类器的个数)

The number of iterations t=0, 1,...T-1: (T is the number of weak classifiers to be selected)

(1) For each feature j, use a single feature to train the classifier h _j , and get the optimal threshold parameter according to the weight W _t of the training sample set, so that the error rate ε _j of h _j is the smallest: ${\mathrm{\&epsiv;}}_j=\underset{i=0}{\overset{\mathrm{no}-1}{\mathrm{\&Sigma;}}}{W}_t\left(i\right)|h_j\left(x_i\right)-{\mathrm{the\; y}}_i|;$

(2) Get the weak classifier with the smallest error rate as the t-th weak classifier h _t of the current layer strong classifier, the corresponding feature number is f _t , and the corresponding error rate is ε _t ;

(3) Calculation parameters ${\mathrm{\&alpha;}}_t=\ln \left(\frac{1-{\mathrm{\&epsiv;}}_t}{{\mathrm{\&epsiv;}}_t}\right),$ ${\mathrm{\&beta;}}_t=\frac{{\mathrm{\&epsiv;}}_t}{1-{\mathrm{\&epsiv;}}_t};$

(4) Update the weights of all samples $W_{t+1}\left(i\right)=\frac{W_t\left(i\right){\mathrm{\&beta;}}_t^{1-e_i}}{Z_t},$ Where e _i =0 means that the current weak classifier correctly identifies the sample x _i , otherwise e _i =1, Z _t is a normalization factor, so that the sum of the updated weights is equal to 1

输出最后的强分类器： 强分类器输出为1

Output the final strong classifier: Strong classifier output is 1

Indicates that the input sample x passes the strong classifier of this layer, otherwise the input sample is considered to be a non-face.

The classification error rate of each layer of strong classifiers for positive samples should be as low as possible. For a single-layer strong classifier, the larger c is, the larger the initial weight of positive samples is, and the smaller the initial weight of negative samples is. The training algorithm is Reducing the error rate ε _t of each weak classifier will ensure that the samples with heavy weights are classified correctly as much as possible. Therefore, the larger c is, the smaller the classification error rate of positive samples is, and the higher the classification error rate of negative samples is. Therefore, we have The training process adopts the method of manually adjusting the parameter c, so that the classification error rate of each layer of strong classifiers for training positive samples is very small, generally less than 0.05%. For single-layer classifiers, we have no clear requirements for the classification accuracy rate of negative samples, that is, the false alarm rate. Because the face detection algorithm is formed by cascading multiple classifiers, the false alarm rate of each layer does not need to be too low, and the total false alarm rate obtained by multiplying the false alarm rates of dozens of layers can be very small. The positive sample error rate of the layer is less than 0.05%, and the total correct rate can still reach 99%, which can ensure that the algorithm can detect various types of training samples, and the training samples already include various types, multiple directions, multiple Therefore, the final implemented face detection model can detect positive face samples with multiple types of interference.

In addition, the number T of weak classifiers of single-layer strong classifiers also needs to be carefully adjusted. The larger T is, the larger the number of weak classifiers is, and the lower the false alarm rate is generally; since each candidate face window is input into the strong classifier, T micro-features corresponding to the strong classifier of this layer need to be calculated first, so T A larger value also means that the computational efficiency of the strong classifier at this layer is higher. During the training process, we need to continuously adjust the size of the classifier T of each layer according to the false alarm rate, and find a compromise between the false alarm rate and computational efficiency. The principle is that the false alarm rate of each classifier should be small, and T should not be too large. Computational efficiency is higher.

In the face detection of the present invention, a two-level face detection algorithm is used to detect and verify the detected faces; the following will be described in conjunction with specific embodiments:

Since the number of micro-features of a 20×20 image reaches 89199, it means that the best features need to be searched out from the 89199 features in each weak classifier training process, and this process is very time-consuming. In order to improve the efficiency of the training algorithm, and also in order to improve the performance of the detection algorithm, the thick and thin two-stage face detection algorithm proposed by the present invention implements the face detection process in two stages, and the face window size of the coarse detection search is fixed at 10× 10. The size of the face window for fine detection and search is 20×20. In the detection process, we also obtain images of different scales, and then halve the resolution of images of each scale, search for face candidate windows with a size of 10×10, and input each window into the strong classifiers of each layer for rough detection , calculate the micro-features of the strong classifiers at each layer, and make a judgment, eliminate the non-face windows, and finally a few candidate face windows are input into the fine detection; expand the 10×10 window that passed the rough detection to a 20×20 window , perform fine detection, and continue to search for these candidate windows in the scale image whose resolution has not been halved, and determine the final face window. Similarly, the training process is also divided into two parts, first training the coarse detection model, and then training the fine detection model.

The resolution of all positive face samples in the training set is halved again to obtain 10×10 rough detection face samples. However, the total number of micro-features in a 10×10 face sample is only 5676, so the training efficiency of the coarse detection model is much higher than that of the fine detection model. In addition, the calculation efficiency of each micro-feature of the face in the 10×10 image is also higher than that of the micro-features in the 20×20 image, so this two-stage detection method can also greatly improve the speed of face detection.

Arbitrarily input an image, in order to detect faces in a certain scale range in the image, we scale the image by scale. For example, if we want to detect faces with scales from 20×20 to 240×240 in a 320×240 image, we need to downscale the image at multiple scales. For coarse detection, the minimum reduction factor should be 2, and the maximum reduction factor should be 24; for fine detection, the minimum reduction factor should be 1, and the maximum reduction factor should be 12. If the ratio of the reduction multiples of two adjacent scales is set to 1.25, the reduction multiples of the image are (for rough detection) 2, 2.5, 3.13, 3.91, 4.88, 6.10, 7.63, 9.54, 11.9, 14.9, 18.6 , 23.3, a total of 12 scales.

Inversely transform the square frames detected by two-level thick and thin in all scale images to the scale and position of the original input image, and obtain the candidate position and candidate size of the face in the original image. In general, a real face is often detected multiple times at different scales and adjacent positions, while the occurrence of false alarms is often relatively isolated. Figure 12a is an example. At this time, it is necessary to post-process the detection results and merge the face frames in adjacent positions. If the size difference and position difference of the two candidate face frames are very small, or the overlapping area of the two face frames is very large, you can Combine these two boxes into one, the position of the combined box is the mean of the positions of the two boxes, and the size is also the mean of the sizes of the two boxes. In the end, a few face frames are left, and each frame is formed by merging a certain number of candidate frames. This value is a very important parameter, which determines whether the detected face frame is a real face. This paper sets a threshold. If the number of merged frames is greater than this threshold, the current position of the face is a real face, otherwise the candidate frame is eliminated. Figure 12b is the combined result.

The face detection algorithm mentioned above can only detect frontal faces, and the face depth changes and plane rotation angles that can be tolerated are very limited. In addition, in order to obtain the face in an image, the face detection algorithm needs to search a large number of zoomed images of different scales. Although the algorithm can detect the face within tens of milliseconds, this process is still very time-consuming and expensive. computationally consuming. If the face detection is performed on each frame of the video sequence input in real time, the calculation amount of the whole algorithm will be very alarming. Another important purpose of face tracking is to achieve continuous tracking of a certain face, to confirm that the target of long-term tracking is the same face, so that subsequent face processing algorithms such as face recognition, expression recognition, etc. can comprehensively recognize multiple frames in the video As a result, the accuracy of the recognition algorithm is greatly improved.

Face tracking is realized on the basis of face detection. Face detection is first performed on the input image of the video sequence. In order to reduce the CPU usage of the program, the face can be detected every few frames. After the face is detected, the detected face is tracked and verified in the next two frames, and the largest face among the verified faces is left, which is continuously tracked and various pose changes of the face are processed. Considering that the skin color of the human face has very distinctive features, which are very different from hair, clothes, and shooting scenes, the present invention also uses the skin color feature to track the face, and the skin color feature is obtained through the color histogram feature. . As shown in Figure 13, this paper calculates the color histogram in the circular area, and uses the face coordinates and histogram features of the k-1th frame to search the image of the kth frame to obtain the face position of the kth frame.

The present invention adopts an object tracking algorithm based on Mean shift and histogram features when performing face tracking. This algorithm uses the histogram features to realize fast tracking of a certain color target. The processing efficiency of the algorithm is very high. Combined with face detection, it realizes the continuous detection and tracking of faces in video sequences.

The method of the present invention can quantize each color space of R, G, and B into 8 levels when calculating the face color histogram feature, and the total color space can be quantized into 8 × 8 × 8 levels, so that each calculation The histogram feature of is 512-dimensional. The face area is described by three parameters (x _cen , y _cen , rad), which respectively represent the xy coordinates of the center point of the face and the radius of the circular face, as shown in Figure 13a. Of course, other spaces can also be used in practical applications Quantitative level implementation. When a new image is input, the tracking algorithm calculates the new position of the face in the current frame based on the size of the face in the previous frame and the histogram features, and updates the radius of the face to reflect the transformation of the face size.

The present invention introduces two local features of long-term histogram and short-term histogram in the process of tracking, wherein the long-term histogram is the mean value of the histogram of the previous dozens of frames, which reflects the changing law of the skin color of the face for a long time, while the short-term histogram The histogram is the average value of the histogram of the previous frames, which reflects the change of the skin color of the face in a short period of time. When searching for the position of the face in the current frame, the matching histogram template used is equal to the mean value of the long-term histogram feature and the short-term histogram feature, so that even if the face in the current frame denies the pose, illumination, expression, etc. The difference between the image templates will not be too large, and the position of the face can be quickly obtained by using the Mean shift algorithm (motion tracking algorithm).

In order to ensure that the tracking target must be a human face, and to ensure that the radius rad can accurately describe the change of the size of the human face, the present invention adds the verification of the tracking area during face tracking, that is, the frontal face detection is performed in the tracking area every few frames, At this time, the position and size of the face are approximately known, so there is no need to search the entire image, only a few positions and a few scales are needed. If a frontal face is detected in the tracking area, the system updates the tracking parameters according to the size and position of the detected face, including the center, radius and histogram features of the face. In addition, if the target is tracked for hundreds of consecutive frames, but no frontal face is detected in the tracking area, it can be considered that the target is not necessarily a human face at this time, and the tracking is stopped. Perform a poor search on the input image in subsequent frames, re-detect the face and track it. This method can be used in face recognition, expression recognition, face synthesis and other methods for real-time detection and tracking.

The face detection algorithm of the present invention can accurately detect faces rotated in a plane from -20° to 20°, and faces rotated in depth from -20° to 20°, and can detect faces with heads up and down in a certain range, and faces with different expressions , wearing or not wearing glasses has no effect on the detection effect. Fig. 14 is a group of face detection results, wherein each face image in Fig. 14a includes a plurality of candidate frames and a processed detection result frame, wherein the frame shown in line 141 represents the post-processed detection result, and line 142 The box shown represents a candidate box, which contains multiple candidate boxes in this figure; the boxes in Figure 14b-d are the output results after post-processing, and the illumination difference between the left and right sides of the face in Figure 14c-d Larger, but the present invention adopts the method of left and right gray scale normalization processing in the method, has improved the ability of detection algorithm anti-illumination interference, has realized the accurate detection of this kind of face, Fig. 15 is a group of tracking results.

The above-mentioned method of the present invention can be realized by the real-time detection and continuous tracking system of human faces in video sequences. With reference to the content of FIG. 16, the system includes a human face detection device 1 and a human face tracking device 2. Unit 11, micro-feature calculation unit 12 and classifier unit 14; described face processing unit 11 receives the image information, zooms the received image, exhaustively searches the candidate face window in the zoomed image, and calculates the window image The mean and variance of the left and right half are then sent to the micro-feature calculation unit 12, and the feature calculation unit 12 calculates the microstructure feature according to the AdaBoost algorithm, and sends it to the classifier unit 14 for Judgment, after the classifier unit 14 makes a decision, it is sent to the human face tracking device 2; the human face tracking device 2 includes an object tracking unit 21 and a tracking area verification unit 22, and the object tracking unit 21 uses the histogram feature to calculate, To realize image tracking, the tracking area verification unit 22 performs area detection on the tracked image, and verifies the tracked face.

In the system of the present invention, the classifier unit 14 of the face detection device 1 also includes a thick and thin two-level face detection unit 13, which receives microstructural features, performs thick and thin two-level detection and post-processing, and determines the face window, Follow up again. The block diagram of this system can refer to the content shown in Figure 17.

The invention realizes the detection and continuous tracking of the human face in the real-time input video sequence. The algorithm adopts the face detection method based on the AdaBoost statistical hierarchical classifier to realize real-time detection of upright faces, and can detect faces with different expressions in different scenes. The face tracking method based on Mean shift and histogram features realizes real-time tracking of detected faces. The speed of the tracking algorithm is fast, the CPU occupancy rate is low, and it is not affected by the posture of the face. Sideways and rotated faces can also be tracked.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention within.

Claims (14)

1, the real-time detection of people's face and the method that continues to follow the tracks of in a kind of video sequence is characterized in that, may further comprise the steps:

Inputted video image;

Adopt the real-time face detection algorithm that the video image of input is carried out the detection of people's face;

Adopt thickness two-stage people face detection algorithm that detected people's face is carried out detection validation again;

Adopt the people's face after the object tracking algorithm keeps track is verified;

By checking people's face of following the tracks of is verified processing to tracing area.

2, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 1 is characterized in that described real-time face detection algorithm is based on the AdaBoost algorithm and is realized by multistage classifier.

3, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 2 is characterized in that described people's face detects and comprises following steps:

The image information that receives is carried out image zoom, searches people's face window;

Detected people's face is carried out positioning feature point, people's face is carried out geometrical normalization;

People's face is carried out the gray balance processing;

People's face is rotated convergent-divergent;

The standard faces image that obtains detecting.

4, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 3, it is characterized in that, comprise following situation in the described step that adopts thickness two-stage people face detection algorithm detected people's face to be carried out detection validation again: as in certain two field picture, detecting one or more people's faces, in ensuing two two field pictures, follow the tracks of these people's faces, and people's face of following the tracks of in follow-up two two field pictures is detected and verifies;

After continuous three frames in certain position detect people's face, determine that people's face exists, select one of them to begin to follow the tracks of.

5, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 3, it is characterized in that, after people's face carried out the gray balance processing, also comprise, the average of half of gray scale about people's face and variance are equated people's face both sides gray scale normalized step respectively.

6, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 3, it is characterized in that, the step that also comprises little feature calculation and sorter judgement behind described people's face rotation convergent-divergent, this step are to adopt the integrogram of the facial image after handling and the microstructure features that square integrogram obtains any yardstick, position in the image.

7, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 4 is characterized in that the described step that detected people's face is verified comprises:

People's face of continue to follow the tracks of selecting in subsequent frame is lower than setting value as the tracking results similarity of back one frame in the consecutive frame and former frame, stops tracking; After last target stops to follow the tracks of, in successive image, carry out people's face again and detect,, verify laggard line trace step up to finding new people's face.

8, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 1, it is characterized in that, the described thickness two-stage people face detection algorithm that adopts again carries out in the step of detection validation detected people's face, the detection of people's face divides two-stage to realize, the resolution of people's face window of rough detection search is surveyed the resolution of people's face window of search less than described examining, image to each yardstick, earlier with after its corresponding dwindling, people's face window with the rough detection search carries out the detection of people's face, eliminate non-face window, in the archeus image, people's face window of surveying search with examining carries out people's face to remaining people's face candidate window and detects again.

9, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 1 is characterized in that in the people's face step after carrying out tracking verification, described object tracking algorithm is based on Mean shift and histogram feature is made.

10, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 9 is characterized in that described histogram comprises long-term histogram, short-term histogram and color histogram.

11, the real-time detection of people's face and the method that continues to follow the tracks of in the video sequence as claimed in claim 9, it is characterized in that, step to the tracing area checking is meant that carrying out people's face every the number frame at tracing area detects, if detect front face at tracing area, then according to the size and the position renewal tracking parameter that detect people's face, described tracking parameter comprises center, radius and the histogram feature of people's face; If continuous hundreds of frame all traces into target, but all do not detect front face, stop to follow the tracks of, in successive image, carry out people's face again and detect,, verify laggard line trace step up to finding new people's face at tracing area.

12, the real-time detection of people's face and lasting tracker in a kind of video sequence is characterized in that, comprise people's face pick-up unit and face tracking device, and people's face pick-up unit comprises people's face processing unit, little feature calculation unit and sorter unit; Described people's face processing unit receives image information, and the image that receives is carried out convergent-divergent, exhaustive search candidate face window, the average of calculation window gray scale and variance; Described little feature calculation unit goes out the microstructure features of each window according to the AdaBoost algorithm computation, and send it to described sorter unit and adjudicate, the sorter unit sends it to face tracking device after adopting thickness two-stage people face detection algorithm to adjudicate; Described face tracking device comprises object tracking unit and tracing area authentication unit, the object tracking unit adopts histogram feature to calculate, realization is followed the tracks of image, and the tracing area authentication unit carries out the zone to the image of following the tracks of and detects, and people's face of following the tracks of is verified.

13, the real-time detection of people's face and lasting tracker in the video sequence as claimed in claim 12, it is characterized in that, the sorter unit of described people's face pick-up unit also comprises thickness two-stage people face detecting unit, receive microstructure features, carrying out the thickness two-stage detects: the resolution of people's face window of rough detection search is surveyed the resolution of people's face window of search less than described examining, image to each yardstick, earlier with after its corresponding dwindling, people's face window with the rough detection search carries out the detection of people's face, eliminate non-face window, in the archeus image, people's face window of surveying search with examining carries out people's face to remaining people's face candidate window and detects, and determines people's face window again.

14, the real-time detection of people's face and lasting tracker in the video sequence as claimed in claim 12 is characterized in that, the histogram that described object tracking unit adopts comprises long-term histogram, short-term histogram and color histogram.

Images