RU2316051C2 - Method and system for automatically checking presence of a living human face in biometric safety systems - Google Patents

Abstract

FIELD: safety and control systems.

SUBSTANCE: as main mechanism, detection of living human face and detection of unsanctioned users present closely to registered user, are used. In invention, methods for tracking a three-dimensional object are used, where object is brought to first normalized face shape, while fast method is used for measuring and comparing facial mimics to a template, and also methods are used for detection of local features and representation of face in three different normalized forms. Also, fast method for measuring and comparing to a template is used for such a behavioral biometric characteristic as phonemic signature, produced as a result of execution of system commands by the user.

EFFECT: increased reliability and speed when detecting an attempt of unsanctioned access to an object.

2 cl, 8 dwg

Description

The claimed invention relates to security and control systems and can be used in the design of automatic access control systems for users in protected areas: premises, buildings, etc.

Modern access control systems use biometric information about a person to identify registered users and deny access to unauthorized users.

One of the features of the development of modern society is the growing need to restrict access to various facilities, for example, offices, warehouses, ATMs, military facilities. This is due to the need to ensure the security of these objects, to prevent theft of intellectual property and goods.

Previously widely used security systems based on the presence of a person, such as a security guard, at the entrance to a protected area, or using video surveillance systems, the data from which are also analyzed by a person, become expensive and do not provide the necessary degree of reliability. It is in connection with these shortcomings in many public and private organizations that systems based on identification magnetic and smart cards have become widespread. The card contains information identifying the user using it. The scanner located at the entrance to the secure room reads this information and makes a decision on providing the user access to the room. A significant drawback of such systems is the ability to use the card by an attacker, for example, he can steal it.

Biometric systems are one of the most promising solutions devoid of these drawbacks. Such systems are based on the analysis of biometric information about the user: facial features, voice, gestures, fingerprints, etc. User biometric parameters automatically read by the system are compared with templates stored in the database. If one of the templates matches the received data, then the user is considered identified and he is allowed access.

Modern biometric systems are based on the analysis of the following biometric parameters of a person: face, voice, iris, gestures. Other types of parameters either do not provide sufficient accuracy for user identification or require contact with a reader (as is the case with fingerprints).

At the same time, the existing biometric systems have a number of drawbacks allowing the possibility of unauthorized access to the protected area.

Known solutions that offer algorithms and control systems based on the analysis of two-dimensional images of a person’s face (see published US patents No. 6,633,655 [1] and No. 6,681,032 [2]). These systems use video cameras to capture face images, realize the allocation of facial areas and calculate their characteristics, which are compared with templates stored in the database. Such systems process 2D models of a person’s face and thereby make it possible to gain unauthorized access by providing cameras with photos of a registered user.

Other well-known solutions (see US patent No. 6,002,782 [3] and published application of the Russian Federation No. 2001122361 [4]) offer methods based on the construction of a three-dimensional model of the human face, and therefore exclude the use of photographs to gain unauthorized access. A vulnerable feature of such systems is the ability of an attacker to gain access to a protected object by using 3D dummy.

In US patents No. 6,567,775 [5] and No. 6,219,640 [6] and in the published application of the Russian Federation No. 2003110435 [7] algorithms are proposed that combine the analysis of video images and their corresponding audio data. The simultaneous use of audio and video information can improve the accuracy of the identification and, therefore, reduce the likelihood of unauthorized entry into the guarded room. However, the proposed methods do not preclude the penetration into the protected area of intruders accompanying the registered user identified by the system.

Closest to the claimed is the solution described in US patent No. 6,498,970 [8], which offers a method and a system for identifying a user using his face, voice and gestures to control access to the car. At the first stage, a 2D image analysis of the person’s face captured by the camera of the system is performed. The data obtained is compared with the templates from the database, and a decision is made to provide access to the car. Further interaction with the car is carried out using voice and gestures. The system compares each supplied command with templates from the database and provides access to the requested function only if the comparison is positive. The main disadvantage of the known solution [8] is its excessive complexity and strict dependence on gestures, although gestures of the same person can vary widely with different health conditions.

The problem to which the invention is directed is to create mechanisms capable of detecting attempts of unauthorized access to an object with a high degree of reliability.

The technical result is achieved due to the fact that the main mechanism used is the detection of a living person’s face and the detection of unauthorized users present next to the registered user.

The proposed method allows to solve the following tasks:

- automatic collection of information about three-dimensional images, speech signals and kinesthetic signals characterizing the observation area;

- performing quick detection, tracking, generalization of information about three-dimensional objects located in the observation zone;

- recognition of a three-dimensional object on the subject of belonging of a person to a living person in the zone of interest by measuring the static and dynamic characteristics of a three-dimensional model of the face, which contain information about the features of the face, its movement, as well as the degree of interactive user interaction with the equipment;

- detection of dummy head;

- identification of active (interacting with the system) users based on a generalization of the results of detection of a living person's face in areas of interest;

- identification of passive (not interacting with the system) users using covert shooting of all objects present in the scene;

- identification of a person according to the physical and behavioral characteristics of a person's face;

- identification of the situation of congestion in the access control system, operating as a queuing system;

- identification of types of criminal situations by analyzing the actions of detected active and passive users and deciding on the type of situation in the observation area in accordance with the application.

The main distinguishing features of the proposed method are:

1) the introduction of additional protection against the recognition of the dummy head;

2) user identification by physical and behavioral biometric characteristics of a person;

3) recognition of users active with the system in an interactive scenario and covert observation of passive users;

4) recognizing the type of situation and identifying a criminal situation in the surveillance zone, identifying the situation of congestion in the channels of the access control system.

Technical implementation of the proposed method includes the following operations:

but. obtaining a three-dimensional image by using three-dimensional sensors, among which at least one sensor interacts directly with the user, and several others can be used for covert observation of the user and surrounding objects;

b. determination of zones of interest by applying the method of projections of three-dimensional and two-dimensional images and analyzing entire image elements, constructing feature maps in probability;

from. application of fast methods of face detection in three different modes, namely: detection, tracking, generalization, as well as the application of the conversion of the detected three-dimensional object to the first normalized face shape;

d. the application of accurate methods for detecting local elements of the face and bringing the first normalized face shape to the second, third and fourth forms, i.e. anthropometric graph, surface, texture;

E. A quick method of measuring and comparing with a behavioral biometric characterization template

f. - signature of the head.

Thus, multi-purpose flexible operation of the system is formed, which, depending on the parameterization of the biometric task and application, for example, accuracy, performance, and other limitations, implements several modes, including:

- recognition mode of a three-dimensional object of the face of a living person;

- face identification mode;

- detection mode of active and passive users;

- a mode for detecting the congestion situation of channels of an access control system operating as a queuing system;

- criminal situation detection mode.

The high performance of this method is achieved by using methods of tracking a three-dimensional object, reduced to the first normalized face shape, using the fast method of measuring and comparing with a template of behavioral biometric characteristics - signature facial expressions. High accuracy and speed of the method is achieved by using accurate methods for detecting local features and representing the face in three different normalized forms, i.e. second, third, fourth. In addition, they use a quick method of measuring and comparing with a template of behavioral biometric characteristics of a phoneme signature obtained as a result of user executing system commands. The sequence of commands is randomly selected, which protects the system from imitation. Commands to the user can be a signal to his action, where actions can be visual, audible or kinesthetic in nature (the action of positioning oneself in front of the camera, pronouncing a phrase, acting on objects). Additionally, a fast method of measurement and comparison with a template of behavioral biometric characteristics is used - a phonemic-mimetic signature obtained as a result of user executing system commands, and a fast method for detecting active and passive faces by summarizing face information in each of the zones of interest. In this case, the method of classifying the situation by type, identifying the criminal situation in the surveillance zone and the situation of the processing channels in the access control system, if used as a mass service system, is used.

The main advantages of the claimed invention are the following factors:

1. resistance of the biometric system to external objects, such as glasses, a beard, a mask, which is achieved by using several three-dimensional sensors, several types of facial representations, and applying detection and generalization of faces from different angles;

2. protection of the biometric recognition system from three-dimensional imitation by analyzing the movements of a three-dimensional object, as well as its parts in accordance with the user's interactive actions, for example, visual, auditory, kinesthetic, against a specific set of system commands and comparing the corresponding static types, in particular, holistic model, anthropometric points, face surface, texture, and dynamic face signatures, i.e. voice, head, facial expressions, phonemes, phoneme-mimic;

3. protection of the biometric recognition system from controlled dummy holograms, which is achieved through the use of a set of interactive commands coming from the command generator block (DBMS) to the display of the system and forcing the user to perform mechanical, i.e. kinesthetic actions on a material object located in the detection zone, for example, to raise an object, press a button;

4. recognition of users active in relation to the system interacting with the system through the use of an interactive scenario for working with various types of sensors;

5. recognition of passive users who are not in contact with the equipment due to covert surveillance;

6. recognition of the situation of congestion of channels access control systems (as a queuing system) to improve the throughput of the biometric system;

7. recognition of a criminal situation and protection of a biometric recognition system from unauthorized access in the event of a registered user entering with unauthorized users or dangerous objects, which is achieved by covert detection of the presence of all possible users or certain types of objects near the head of a living person;

8. the use of static and dynamic signatures of both physical and behavioral biometric characteristics for face recognition.

The claimed invention includes a method and system for automatically checking whether a three-dimensional object in the detection zone is the face of a living person.

The proposed method and system are illustrated by illustrations in figures 1-8, which, in particular, shows a possible appearance of the system:

figure 1 presents a diagram of the interaction of various elements of the entire system and individual devices for detection, normalization, recognition, decision making;

figure 2 shows an example of a visual message to the user, i.e. positioning, in accordance with visual commands;

figure 3 shows an example of an audio message to the user in accordance with the commands;

4 gives an example of detecting an interactive and passive user through covert surveillance;

5 gives an example of a first normalized face shape, i.e. holistic face model;

Fig.6 shows an example of a second normalized face shape, i.e. anthropometric face graph;

Fig. 7 gives an example of a third normalized face shape, i.e. color segmentation of the texture of a three-dimensional face image;

Fig. 8 illustrates an example of a fourth normalized face shape, i.e. surface model.

This system is composed of the following elements: - a system for detecting three-dimensional objects and checking them for whether a person belongs to a living person in biometric systems, consisting of:

1) a block of active sensors, including three-dimensional image sensors, a stereo microphone, a kinesthetic sensor, such as a keyboard, tablet, and display, which are placed on the stage for direct interactive work with the user; the active sensors block is a set of sensors for entering information of various types, in particular, three-dimensional images, audio signals, kinesthetic signals, about the observation area; active 3D cameras or passive stereoscopic systems can be used to enter a three-dimensional image;

2) a block of passive sensors, which includes sensors of three-dimensional images, which are placed on the stage for covert observation of users;

3) a block for capturing and pre-processing the image signals, voice or kinesthetic signal;

4) a memory unit for recording frames of three-dimensional images;

5) a device for detecting a three-dimensional object and primary normalization, consisting of blocks for capturing a zone of interest, detection, tracking, generalization, which can be implemented on separate processors, and a shared memory block into which information about a face-like three-dimensional object is recorded and read to the first normalized face shape;

6) devices for normalizing the shapes of a three-dimensional object, consisting of blocks for normalizing a three-dimensional object, an anthropometric analysis block, texture analysis, surface analysis and a shared memory block, in which information about a face-like three-dimensional object is written to and read from, reduced to the second, third and fourth normalized face forms and static signature generation block;

7) devices for normalizing the movement of a three-dimensional object, consisting of blocks for recognizing the movement of the head, recognition of facial expressions, recognition of phonemes, synchronization of phonemes and facial expressions, recognition of kinesthetic signals and blocks of shared memory, in which information about the signature of the head and voice signature is recorded and read, mimic signature, phoneme-mimic signature, kinesthetic signature;

8) a decision-making device, consisting of a database management system consisting of commands, signature standards, comparison metrics, recognition ratings, a database consisting of commands, signature standards, comparison metrics, face recognition ratings, a decision block for recognizing a person’s identity to a living person in each of the zones of interest, the unit for counting active and passive persons, the unit for identifying the person, the unit for recognizing a criminal situation.

- stereo cameras, which are two or more cameras, acting as optical sensors, spaced a certain distance and calibrated relative to each other; while the collection of audio signals from the user is carried out by using a stereo microphone that captures the signal through two channels, which allows for additional cleaning of the signal from noise, and the collection of kinesthetic signals from the user is carried out using a keyboard or any other sensor that require physical contact with the equipment.

As the simplest configuration of a system for protection from dummy holograms, only a signal exceeding a certain threshold is detected. For better protection against a controlled hologram, it is advisable to use a device that measures the weight or height of a user, a device that measures a person’s signature with a pen, a device that reads a pin code typed on the keyboard. As a sensor for capturing an audio signal, such a microphone or stereo microphone can be used, which allows for additional noise cleaning. This unit is placed in close proximity to a user undergoing biometric control, for the purpose of interactive user operation with the system. An active user receives commands from the system using a display on which interactive messages are issued to the user to perform visual actions, for example, “position the face and look into the camera”, auditory, for example, “say something”, and kinesthetic, for example, “perform mechanical act". The scenario of a user working with the system can be fixed or it can be chosen in some random way in order to further protect the system from imitation. An example of an active scenario is shown in figure 2 and figure 3.

The passive sensor block is a set of three-dimensional sensors for inputting a three-dimensional image.

The number and relative orientation of the sensors is chosen in such a way as to ensure the shooting of the entire perimeter of the observation zone. In particular, some active sensors of three-dimensional images can work in the search mode of passive users. As the minimum configuration of the system of active and passive sensors - choose one three-dimensional sensor that works in two search modes for active and passive users.

The blocks for capturing and pre-processing the image signals, voice and kinesthetic signal capture the signal, clean the signal from noise and preprocess the signal, in particular, for local normalization of brightness, in a convenient representation of the signal for recognition methods.

The blocks of the buffer memory store the frames of three-dimensional images coming from the sensors of three-dimensional images, as well as information about areas of interest in the observation zone.

The zone of interest capture unit evaluates the observation zone and determines a plurality of zones of interest. To do this, use the block to carry out the following steps:

1) carry out stereo reconstruction of the entire observation zone, i.e. constructing a disparity map characterizing the remoteness of the observation objects from the camera, which is the main characteristic of a three-dimensional image;

2) construct projections of a three-dimensional image in directions, i.e. select two-dimensional images;

3) carry out image filtering, for example, using gradient low-pass filters, Gabor filters, color segmentation;

4) further by the whole image analysis procedures, i.e. holistically, they determine the global characteristics of the region of interest, such as the predominance of facial tones, three-dimensional objects, horizontal and vertical borders of the face, the presence of clusters of local features of the face or the whole face. Whole image analysis procedures can be, for example, projection methods and pattern matching methods. Figure 5-7 shows an example of stereo reconstruction, color segmentation, Gabor transform;

5) according to the estimates of the characteristics of the three-dimensional image on this and the previous frame, they decide on the importance of the zone of interest and set the priority.

In the detection unit, the analysis of trait maps is performed as follows:

1) the pyramid of two-dimensional images is specified based on the obtained calculated disparity map;

2) divide three-dimensional information into two sets: a set of projections - two-dimensional images and a lot of disparity maps; in many two-dimensional images, color information is additionally highlighted, that is, three sets of feature maps are selected that are responsible for the representativeness and information content of the face model. Based on this information, normalized forms are subsequently built;

3) choose a feature map of images, in particular a color map, a Gabor map, a disparity map;

4) clustering of attributes by brightness, tone or disparity, by the size of regions of interest, using, in particular, the K-means method, Kohonen neural networks or radial basis networks;

5) isolate connected components and local objects;

6) based on the geometry of the face, analyze the location of local features and find the face.

When forming a decision on face detection, at least one of three representations of face models is used:

a) representation using local traits, clustered traits: eyebrows and eyes; nostrils, mustache, mouth, chin; cheeks and nose;

b) color segmentation representation;

c) presentation by analysis of a disparity map — an analysis of the convexity of a plurality of corresponding points, in a number of sources called “correspondence points”.

Using the tracking unit in the zone of interest, the speed of the detection unit is increased by using information about the previous frame and the movement of the object. To do this, perform the following sequence of steps:

1) select a set of reference points for representing the face in one of three forms characterizing the face in the already found frame of a three-dimensional image;

2) analyze the optical flow of many points and calculate the motion parameters;

3) in the next frame, they search for the most probable corresponding point in the local neighborhood of the reference point;

4) check the consistency of points - if the object is detected with a low degree of confidence, start the procedure from the detection unit.

Using the generalization unit, the accuracy of the detection unit is increased by using multiple frames of three-dimensional images as follows:

1) select a set of reference points for representing the face in one of three forms characterizing the face in the already found frame of a three-dimensional image;

2) analyze the optical flow of multiple points of three-dimensional images, built on a series of several frames;

3) undetected points and points found with low probability are specified by reconstruction procedures and checks of a normalized person for consistency and information content.

A face-like three-dimensional object is a shared memory with which the face detection process works on each frame. Data on a three-dimensional image and a face is stored in a first normalized face shape. The first normalized form describes:

1) holistic holistic information about the face, in particular the center of gravity of the head, the dimensions of a three-dimensional object;

2) representation of the face as a set of clusters of traits, for example, a face as a set of three clusters of “eyes / eyebrows”, “nostrils / mustache / lips / chin”, “cheeks / nose”.

3) representation of a person in the form of a three-dimensional object characterized by a disparity map;

4) representation of the face in the form of a color card that determines the tone of the skin of the face.

A device for recognizing and normalizing static forms of a three-dimensional object consists of:

- groups of blocks for checking the static elements of a three-dimensional object for the presence of a face of a living person in the detection zone. At the input, the device receives a three-dimensional object in its first normal form, then the object is sequentially refined with three different representations of the face, namely, the graph of anthropometric features of the object, the texture of the object, the surface of the object, and reduced to the second, third and fourth normalized face shapes, respectively.

Normalized forms differ in the degree of accuracy, reliability and speed of construction. Using the obtained normalized forms, a static signature of the person is built, which, depending on the application parameters, for example, weight of importance, degree of reduction, can be used as a biometric characteristic for determining the face of a living person or identifying a person by face. Another application of normalized forms is their use for constructing various models of face movement and generating a dynamic face signature from them, which includes voice signature, head signature, facial expression, phoneme-facial signature, kinesthetic signature.

Using the anthropometric analysis unit, the following actions are performed:

1) the construction of heuristic patterns in the zone of interest, for example, by color segmentation;

2) the choice of the pyramid of images in accordance with the remoteness of the object from the camera, the so-called disparity map;

3) the construction of many two-dimensional images on the projections of a three-dimensional image in accordance with the image pyramid;

4) filtering by gradient filters and Gabor filters depending on the chosen direction of designing a three-dimensional image and a pyramid of images;

5) building representative and informative maps for the face and local features;

6) clustering of features in the current frame, as a result of which the brightness of the image of local features is normalized;

7) identification of a person’s object based on expert heuristic rules;

8) refinement of all anthropometric facial features with accurate facial detection algorithms, in particular, methods of active and deformable patterns;

9) detection of the corresponding points on the feature maps, the disparity map is specified and the anthropometric facial features are verified by comparison with the updated disparity map;

10) verification of the model of the face for consistency by calculating three-dimensional coordinates and distances for anthropometric points;

11) reconstruction of the face model in the form of a three-dimensional graph;

12) the normalization of a three-dimensional graph and the construction of a second normalized shape.

The second normalized form describes the following information:

1) two-dimensional information about the person in the form of a set of anthropometric points and distances determined by the methods of forensic identification;

2) three-dimensional information about local features;

3) a set of face graphs from different angles.

In the block analysis of the texture carry out:

1) the construction of many two-dimensional images from the projections of a three-dimensional image;

2) the representation of images in the HSB model from RGB and the selection of the first component of the tone (hue);

3) threshold filtering or color segmentation (for example, color segmentation according to the K-means method);

4) selection of connected components by tone and detection of local objects;

5) bringing the face model into a third normalized form.

In the third normalized form, the following information is described:

1) local signs of surfaces by color combinations;

2) differences in tone and brightness.

One of the main purposes of the texture analysis unit is to obtain heuristics to speed up the process of detecting facial features.

In the block analysis of surfaces carry out:

1) the construction of a subset of the disparity map, where the corresponding points are found with low probability, on these subsets using the projection method build two-dimensional images, refine the corresponding points, reconstruct and smooth three-dimensional surfaces;

2) the allocation of the basic supporting surfaces of the face and three-dimensional face elements corresponding to the fixed part of the face;

3) the construction of three-dimensional objects on the surface of the face;

4) finding features of a three-dimensional object (integral characteristics, characteristics according to surface levels, convexity);

5) bringing the face model into the fourth normalized form;

6) in the case of the presence of external objects (glasses, hat, mustache), clarification of the fourth normalized form based on the second and third.

The fourth normalized form describes:

1) the type of three-dimensional shape and its features;

2) the type of level lines and its signs;

3) the convexity of the surface and its signs.

In the feature reduction block, the most important features for the application are selected, that is, they carry out the feature reduction procedure for all normalized forms, determine the sample, and establish the importance weights of the features in this sample. These parameters are selected depending on the biometric application: face identification, detection of a living person’s face and dummy, search for active / passive users, recognition of a criminal situation.

The device for recognizing and normalizing the dynamic forms of a three-dimensional object consists of a group of blocks for recognizing sound, recognizing a kinesthetic signal, and recognizing the movement of elements of a three-dimensional object for the presence of a living person's face in the detection zone. To recognize the movement, one of the normalized forms of the face is used. The movement of a three-dimensional object is sequentially refined by applying normalization and signature building procedures: voice signature, head signature, facial expression, phoneme-mimic and kinesthetic signature.

Using the head movement recognition unit, the following steps are performed:

1) the choice of tracking points on a three-dimensional object, for this they use the first normalized face shape and such signs as the center of gravity of the head, reference points of the face that characterize the fixed parts of the face;

2) tracking objects - the construction of the trajectory of the center of the head and the trajectory of the plane of the face, characterizing the focus and gaze of the face, where the trajectory of movement is built as an adequate result of the user's interactive work with the system in accordance with a random scenario;

3) discretization of the trajectory according to the main components and the choice of the main states, where the type of state is determined by the type of head angle, that is, the spatial orientation of the head;

4) construction of a graph of transitions of one state of a person to another;

5) refinement of motion parameters, such as speed, acceleration;

6) the construction of the signature of the head, which is a set of states and transitions in accordance with the scenario template and can be implemented using a graph, a finite state machine, hidden Markov models, etc.

The following steps are performed in the phoneme recognition unit:

1) elimination of noise in the signal;

2) signal segmentation in areas of interest;

3) signal discretization;

4) phoneme recognition by extracting the local-frequency properties of the signal;

5) clarification of speech parameters, such as intensity, volume, tempo, intonation;

6) the construction of a speech signature, which is a set of states and transitions in accordance with the scenario template and can be implemented using a graph, a finite state machine, hidden Markov models, etc.

In the block recognition of kinesthetic signals carry out:

1) detection of a signal about the user's contact with the equipment;

2) filtering the signal by the threshold;

3) making a decision on a dummy-hologram or detection of an active user face.

Using the recognition unit facial expressions produce the following steps:

1) the choice of local objects on a three-dimensional object, for this use normalized face shapes No. 2, No. 3, No. 4 and such elements of the head as eyebrows, eyes, lips, jaw;

2) tracking of three-dimensional local objects - construct the motion paths of local objects, where the motion path is an adequate result of the user's interactive work with the system in accordance with a random scenario;

3) discretization of the trajectory along the main components and the choice of ground states, where the types of states are determined, for example, as follows:

a) eyebrows raised / lowered;

b) the eye is wide open / lowered eyelid / closed;

c) mouth (open / closed / stretched into a smile);

d) the jaw is tightened / lowered / beveled left / right / extended forward;

4) construction of a graph of transitions of one state of a person to another;

5) refinement of motion parameters, such as speed, acceleration;

6) the construction of the trajectory of facial expressions, where each facial state is determined by the states and transitions of local features, such as eyebrows, eyes, mouth, jaw;

7) discretization of the trajectory according to the main components and the choice of the basic states of facial expressions;

8) building a facial expression, which is a set of facial expressions and transitions in accordance with the scenario template and can be implemented using a graph, a state machine, hidden Markov models, etc.

Using the block synchronization of phonemes and facial expressions

1) they build the intersection of sets of facial states with a lot of phoneme states;

2) choose the most probable state transitions;

3) they build a phoneme-mimic signature, which is a set of phoneme and facial expression states and transitions in accordance with the scenario template and can be implemented using a graph, a finite state machine, hidden Markov models, etc.

In the feature reduction block, the most important dynamic signatures for the application are selected, that is, the signature reduction procedure is carried out, the selection is determined, and the importance weights of the signatures in the given selection are established. These parameters are selected depending on the biometric application: face identification, face and dummy detection, search for active / passive users, recognition of a criminal situation or a situation of congestion of access control system channels.

The decision-making device consists of the following blocks: a database of commands and standards, a database management system, a face recognition module of a living person, a block for counting active and passive faces, a face identification block, a situation identification block.

The database is a table in which information about the type of application, scripts of commands, commands to the user, stored standards of face signatures, registered patterns of biometric characteristics, patterns of situations, comparison metrics are placed.

Using the database management system, tables are formed, data is sampled on demand, samples of comparison metrics are made, signatures are compared with standards and a solution is formed depending on the type of application.

In the recognition unit of a person’s face to a living person, the recognized static and dynamic face signatures are compared with the reference ones, a generalized assessment of the presence of a person in a given zone of interest is made and, depending on the results, a decision is made on whether the person belongs to a living user.

In the block of counting active / passive faces, information about the observation area is compiled for all areas of interest, the results of recognizing whether a person belongs to a living person are specified, determining the type of user active or passive, and the presence of models in the observation area is also identified, for example, a two-dimensional fake, three-dimensional fake, hologram.

Using a face identification unit, physical and behavioral biometric characteristics are measured and compared with standards.

Using the situation identification unit, a situation is classified according to its type depending on the number of active and passive persons, the results of recognition of a person's belonging to a living person, and the geometric position of persons in the space of the observation zone. In the simplest case, the degree of criminality of the situation is assessed and the criminal situation or normal is determined in accordance with the threshold. A decision on the situation in the observation zone can also be made by analyzing changes in the current situation - changes in the number of active and passive persons in the detection zone. A similar block is relevant if this system is used to identify traffic jams and channel congestion in access control systems, in this case, queuing systems. Using this unit, it is possible to control the throughput of the biometric recognition system.

Devices 1, 2, 3, 4 (Scheme 1) can be implemented independently on a separate processor, for example, a digital signal processor.

The preferred option for implementing the system is its implementation in other security security systems, including biometric ones, in order to increase the reliability of the system using technologies: protection from dummy, additional protection against unauthorized access, active tracking of an interactive user and covert surveillance of passive users, detecting criminal situations and situations of congestion in the surveillance zone.

The scope of this technology is mainly focused on additional protection of the biometric security system in solving the following problems:

- the task of distinguishing between a face of a living person and a face-like dummy (minimizes the detection error of non-living individuals), namely:

1) protection of the security system from two-dimensional models of the face, for example, photographs. This problem is solved by using three-dimensional sensors;

2) protection of the security system from three-dimensional models of the face. This problem is solved by using the command generator to give the user’s actions an unpredictable and interactive character, i.e. use of a pseudo-random scenario;

3) protection of the security system from three-dimensional holograms, i.e. from an inactive user. This problem is solved by using other types of sensors that measure not visual, audio signals from the user, but, for example, direct kinesthetic contact with the keyboard or other objects, a thermogram, etc.

- recognition tasks of the type of situation in the observation zone:

1. recognition of a criminal situation in biometric access control systems, machine vision and robot vision systems, video surveillance systems, in human-machine interface systems.

A criminal situation is identified in the following cases:

- if, in addition to the active user interacting with the biometric access control system, the system recognizes passive users in the surveillance area. For example, a user withdraws money from an ATM, and two other unauthorized users are behind him);

- in case of recognition by the active user of the system of dangerous objects, for example, weapons;

- if the prescribed number of persons is not found in the observation area of the biometric system, for example, two guards are required to access the bank safe.

2. Recognition of the situation of channel congestion in access control and queuing systems:

- recognition of a situation of crowds of users. Such recognition is important to improve access control system throughput. This indicator is especially important if the access control system works as a mass customer service system. In such tasks, flexible automatic adjustment of the false-acceptance rate (FAR) and false-rejection rate FRR errors of the biometric system is important.

User identification task:

The method allows to increase accuracy indicators (minimize the false-acceptance rate error rate) and access control system performance by integrating other types of biometric characteristics, namely:

1. Static signature of a person;

2. The dynamic signature of the movement of the head, which is the trajectory of the three-dimensional object, reduced to the first normalized face shape;

3. A dynamic mimic signature, which is a finite state machine in which states are identified by the normalized face shapes No. 2-4, and the transitions are the trajectories of the face;

4. A dynamic phoneme signature, which is built by recognizing a specific phoneme alphabet;

5. Dynamic phonemic-mimic signature, which is a combination of phoneme and mimic signatures.

It should be noted that the system can be used not only to protect security access control systems, but also in the following applications:

1) intelligent video surveillance - the system records the surveillance zone and recognizes the criminal situation in crowded situations. For example, a street is shot, while it is assumed that the system captures persons who are passive with respect to user equipment. The following case is possible: for example, if two more people approach one person on the street, a signal is sent about a possible criminal situation;

2) human-machine interface. For example, a person with physical disabilities (disabled person) can interactively work with a computer, controlling it with the help of head movement;

3) vision of robots;

4) video conferencing;

5) computer animation and games.

The system in the most simplified version is implemented in the presence of three-dimensional image sensors, a microphone and a device board, which includes at least one signal processor.

Claims (24)

1. A method for automatically checking the presence of a living person in biometric systems, consisting of the following operations: remote collection of three-dimensional image data characterizing the observation area, which is based on the use of a three-dimensional sensor, in the form of a stereoscopic system consisting of two or more optical sensors with a predetermined location; carry out remote collection of audio signals about the observation area, while using a stereo microphone to obtain a signal; collecting data about the active user according to the results of the user's interactive mode of operation with the system according to a scenario that includes visual, auditory, kinesthetic user actions and is set in some random way; collect data about the passive user based on the results of covert detection of three-dimensional objects by other sensors; capture zones of interest within the observation zone based on an analysis of global image characteristics; carry out three-dimensional detection of the elements of the object in each of the captured zones, using a sequential refinement of the characteristic features of a person; tracking the detected features of the features of the object in the subsequent three-dimensional image; generalize features of the object based on a sequence of three-dimensional images; bringing the image of the three-dimensional object to the first normalized face shape having only global characteristics; they check the static elements of a three-dimensional object for the presence of a face in the observation zone, during which a three-dimensional object in the first normalized form is sequentially refined by at least one of three different representations of the face, namely, the graph of anthropometric features of the object, the texture of the object, and the surface texture an object that corresponds to the second, third and fourth normalized forms of representation of a person; they check the dynamic characteristics of a three-dimensional object, such as sound and motion parameters of elements of an object reduced to one of four normalized forms of face representation, in order to determine whether this object is a living person’s face, while the movement of a three-dimensional object is subsequently refined using normalization procedures and construction of signatures, namely, voice signature, head signature, facial expression, phoneme-mimic and kinesthetic signature; detecting the face of a living person based on the analysis of static and dynamic signatures in each zone of interest, analysis of the results from the selected zones of interest, which are many active and hidden zones of interest, counting the number of people and deciding on the situation in the observation zone. 2. The method according to claim 1, characterized in that the step of capturing multiple areas of interest, the proposed location of the head and face of the user, as well as setting priority to areas of interest, is carried out by performing the following operations: construction and analysis of a disparity map characterizing the remoteness of the observation objects from the camera, performing color segmentation and filtering procedures for a set of Gabor filters to clarify the disparity map; calculation of location coordinates and sizes of zones of interest based on projection methods and pattern matching methods; prioritization of viewing areas in accordance with estimates on the current and previous sets of frames; 3. The method according to claim 1, characterized in that the construction of the first normalized form of representation of the face is carried out using the detection procedure in the zone of interest, consisting of the following operations: the specification of the pyramid of two-dimensional images based on the obtained calculated disparity map; the division of three-dimensional information into two sets, namely, a set of projections - two-dimensional images and a plurality of disparity maps; highlighting color information in a two-dimensional image and performing color segmentation; filtering two-dimensional images by a set of Gabor filters and using clustering; the decomposition of three-dimensional information into three sets of feature maps responsible for the representativeness and informativeness of face information; the use of trait detection procedures on trait maps and the construction of three representations of the face model; representation and detection of the face with the help of local features combined in such clusters of features as “eyebrows / eyes”, “nostrils / mustache / mouth / chin”, “cheeks / nose”; face representation and detection using color segmentation; representation and detection of a face by analyzing a disparity map in the form of an analysis of the convexity of a plurality of corresponding points. 4. The method according to claim 1, characterized in that the construction of a three-dimensional object, reduced to the first normalized face shape, is carried out by the following tracking procedure in the zone of interest, which improves the speed of detection: use the division of the set into three representations of the face; analyze the optical flow of multiple points; points not found are specified by detection procedures; in case of poor tracking, the detection procedure is started. 5. The method according to claim 1, characterized in that the construction of a three-dimensional object, reduced to the first normalized face shape, is carried out using the following generalization procedure in the zone of interest, which improves the accuracy of the detection of multiple frames of three-dimensional images: divide the set into three representations of the face; analyze the optical flow of many points of three-dimensional images, built on a series of several frames; specify undetected points and points found with low probability using reconstruction procedures and checking a normalized person for integrity and information content. 6. The method according to claim 5, characterized in that in order to build a second normalized form, the following steps are carried out: constructing a plurality of two-dimensional images from projections of a three-dimensional image; use filtering by gradient filters and Gabor filters depending on the chosen direction of designing a three-dimensional image; construct representative and informative maps of local features; cluster features on the frame and normalize the brightness of the image of local features; identify the entity of an entity based on expert heuristic rules; refine anthropometric facial features with accurate detection algorithms; find points of correspondence, refine the disparity map and verify the anthropometric facial features by comparing it with the updated disparity map. 7. The method according to claim 5, characterized in that in order to build a third normalized form, the following steps are performed: constructing a plurality of two-dimensional images from projections of a three-dimensional image; represent the images in the HSB model from RGB and select the first component of the tone; use threshold filtering; construct representative and informative maps of local features; clustering features on the frame and normalizing the brightness of the image of local features; identify the entity of an entity based on expert heuristic rules; refine anthropometric facial features with accurate detection algorithms; find points of correspondence, refine the disparity map, and anthropometric facial features are verified by comparing it with the updated disparity map. 8. The method according to claim 5, characterized in that for the construction of the fourth normalized form, the following steps are carried out: build subsets of the disparity map, where the match points are found with a low probability, on these subsets using the projection method build two-dimensional images, refine the match points, reconstruct and smooth three-dimensional surfaces; identify the main surface of the face and three-dimensional elements of the face, after which they find the integral characteristics of objects and surfaces; analyze the convexity of three-dimensional shapes; in the case of the presence of external objects, such as glasses, hat, mustache, refine the fourth normalized form based on the second and third. 9. The method according to claim 5, characterized in that to construct a static signature of the person, the procedure of reducing the signs of the first, second, third and fourth normalized forms and weight of the signs is selected depending on the biometric application, namely face identification, face detection and dummy, recognition of a criminal situation, recognition of a crowd of users, i.e. traffic jams. 10. The method according to claim 1, characterized in that for dynamically checking a three-dimensional object for a face object, a living person uses at least one of the blocks: a head signature block, a voice signature block, a mimic signature block, a phonemic mimic signature block, a kinesthetic block signatures, block for reducing features into a dynamic signature. 11. The method according to claim 10, characterized in that to obtain a dynamic signature of the head, the trajectory of the three-dimensional object reduced to the first normalized form is used, while the trajectory is constructed as an adequate result of the user's interactive work with the system in accordance with a random scenario and To construct the trajectory, use points such as the center of mass of the head and fixed points on the surface of the face that determine the look and focus of the face. 12. The method according to claim 10, characterized in that to obtain a voice signature using segmentation of the audio signal and recognition of phonemes by extracting the local-frequency properties of the signal. 13. The method according to claim 10, characterized in that the second, third, fourth normalized forms are used for mimic signature, the procedures for accurate detection and tracking of local features, which are eyebrows, eyes, mouth, chin, normalize the parameters of their movement and build deterministic finite state machine for describing the facial expression state. 14. The method according to claim 10, characterized in that to obtain a phonemic-mimic signature, the mimic and speech signature is synchronized and one automaton is built on the basis of two finite state machines, which allows detecting facial expressions with higher accuracy. 15. The method according to claim 10, characterized in that for protection against a controlled hologram and obtaining a kinesthetic signature, which is a measure of contact with the equipment, a keyboard is used with which the user types a PIN code. 16. The method according to claim 10, characterized in that to obtain a kinesthetic signature using a touch tablet on which the user puts his signature. 17. The method according to claim 10, characterized in that to obtain a kinesthetic signature using a physical device that measures the weight of the user. 18. The method according to claim 10, characterized in that to obtain the kinesthetic signature using a physical device that measures the growth of the user. 19. The method according to claim 10, characterized in that the dynamic signature of the person is obtained by the procedure of reducing the signs of the first, second, third, fourth normalized forms and the weight of the signs is selected depending on the biometric application. 20. The method according to claim 1, characterized in that the decision on the situation in the observation area is made according to the identification of the active person in accordance with the static and dynamic signatures of the person. 21. The method according to claim 1, characterized in that the decision on the situation in the observation area is made according to the results of identification of passive persons in accordance with their static and dynamic signatures of a person where passive persons who do not interact with the equipment are identified by covert observation. 22. The method according to claim 1, characterized in that the decision on the situation in the observation area is made by analyzing the changes in the current situation and tracking the number of active and passive persons in the observation area. 23. The method according to claim 1, characterized in that the decision on recognition of the face from the dummy hologram is made by analyzing mechanical changes in the current situation and checking the contact of the active user with the kinesthetic sensor, made primarily in the form of a keyboard, scales, tablet for signatures. 24. The system for detecting three-dimensional objects and checking them for the presence of a living person’s face in biometric systems, consisting of the following components: an active sensor unit, including three-dimensional image sensors, a stereo microphone, a kinesthetic sensor and a display, which are placed on the stage for direct interactive work with the user; a block of passive sensors, including sensors of three-dimensional images, which are placed on the stage for covert observation of users; a block for capturing and pre-processing the image signals, voice or kinesthetic signal coming from the block of active sensors and the block of passive sensors; a memory unit for recording frames of three-dimensional images coming from coming from a block of active sensors and a block of passive sensors; a device for detecting a three-dimensional object and primary normalization, consisting of blocks for capturing an area of interest, a detecting unit, a tracking unit, a generalization unit, while the unit for capturing a zone of interest is intended for processing the stored signals of three-dimensional images, and the detection unit, the tracking unit and the generalization unit are intended for processing signals from the block of capture of the zone of interest and storing and reading the results of this processing as information on a face-like three-dimensional object, reduced to the first normal the called shape of the face; a recognition device and normalization of the static forms of a three-dimensional object, designed to process information of a three-dimensional object, reduced to the first normalized form, and consisting of a normalization block of a three-dimensional object, an anthropometric analysis unit, a texture analysis unit, a surface analysis unit and a shared memory unit, which is intended for recording and reading information about a face-like three-dimensional object, reduced to a second, third, fourth normalized face shape, and a reduction unit and a gene talkie static signatures intended for processing signals anthropometric analysis unit, the analysis unit and the surface texture analysis unit; a device for recognizing and normalizing the dynamic forms of a three-dimensional object, consisting of a head movement recognition unit, a facial expression recognition unit, a phoneme recognition unit, a phoneme and facial expression recognition unit, a kinesthetic signal recognition unit, shared memory units that are designed to write and read signature information head, about voice signature, facial expression, phonemic signature, kinesthetic signature, as well as from the block of reduction and generation of dynamic signature, The values for recognition processing block signals facial expression, phoneme recognition unit, the synchronization unit phonemes and facial expression, kinesthetic signals recognition unit: decision making device consisting of database management systems; a database intended for storing teams, signature standards, comparison metrics, face recognition ratings; a decision-making unit on whether a person belongs to a living person in each of the zones of interest based on an analysis of the results of contact of the active user with the kinesthetic sensor; a block for counting active and passive faces, where the separation of these two types of objects is carried out on the basis of the analysis of static and dynamic signatures of a face received from a recognition device and normalization of static forms of a three-dimensional object and a recognition device and normalization of dynamic forms of a three-dimensional object; a face identification unit for interacting with a database management system for comparing a user's biometric characteristics with reference ones; a situation type recognition unit, designed to classify the type of situation on the stage based on the results from the unit for counting active and passive faces and by analyzing the mutual geometric position of the persons in the observation area.

Images