CN108052813A - Unlocking method and device of terminal equipment and mobile terminal - Google Patents

Abstract

The application discloses an unlocking method and device of terminal equipment and a mobile terminal, wherein the method comprises the following steps: when the unlocking operation of a user on the terminal equipment is detected, obtaining voiceprint information and face three-dimensional information of the current user, wherein the face three-dimensional information is obtained by using structured light; judging whether the current user is legal or not according to a preset voiceprint library and a preset human face three-dimensional information library respectively; and if the terminal equipment is legal, unlocking the terminal equipment. According to the method, the voiceprint information and the face three-dimensional information of the current user are simultaneously identified to verify whether the user is a legal user, only the legal user can unlock the terminal equipment, and compared with the method of only carrying out voiceprint unlocking or only carrying out face unlocking, the method can better guarantee the use safety of the terminal equipment, and further improve the user experience.

Description

Unlocking method and device of terminal equipment and mobile terminal

technical field

The present application relates to the technical field of communications, and in particular to a terminal device unlocking method, device and mobile terminal.

Background technique

With the development of science and technology, terminal devices such as mobile phones and tablet computers have increasingly powerful functions, and terminal devices have increasingly become one of the important tools in people's daily life and work.

At present, most terminal devices have the function of screen unlocking. When the screen of the terminal device is locked, when the user wants to use the terminal device, the legality of the user identity needs to be verified first. Only verified users can unlock the terminal. The screen of the device and use the terminal equipment, so as to ensure the safety of the terminal equipment.

The more common unlocking methods include sliding screen unlocking, pattern unlocking, fingerprint unlocking, voice unlocking, and face unlocking. However, a single unlocking method is still lacking in ensuring the security of use of the terminal device. Therefore, how to better ensure the security of use of the terminal device has become an urgent technical problem to be solved.

Contents of the invention

The purpose of this application is to solve one of the above-mentioned technical problems at least to a certain extent.

For this reason, the first purpose of this application is to propose a method for unlocking a terminal device. This method verifies whether the user is a legitimate user by simultaneously identifying the voiceprint information and three-dimensional face information of the current user. Only legitimate users can Unlocking the terminal device can better ensure the security of the terminal device and improve the user experience compared to only unlocking the voiceprint or face unlocking.

The second purpose of the present application is to propose an unlocking device for a terminal device.

The third object of the present application is to provide a computer-readable storage medium.

The fourth object of the present application is to provide a mobile terminal.

A fifth object of the present application is to propose a computer program.

The method for unlocking a terminal device according to the embodiment of the first aspect of the present application includes: when it is detected that the user performs an unlocking operation on the terminal device, acquiring the current user's voiceprint information and facial three-dimensional information, wherein the facial three-dimensional information is obtained using Acquired by structured light; judge whether the current user is legal according to the preset voiceprint database and the preset three-dimensional face information database; if legal, unlock the terminal device.

According to the method for unlocking a terminal device according to an embodiment of the present application, when it is detected that the user performs an unlocking operation on the terminal device, the current user's voiceprint information and facial three-dimensional information are obtained, wherein the facial three-dimensional information is obtained by using structured light ; According to the preset voiceprint database and the preset three-dimensional face information database respectively, it is judged whether the current user is legal; if legal, the terminal device is unlocked. This method simultaneously identifies the current user's voiceprint information and facial three-dimensional information to verify whether the user is a legitimate user. Only legitimate users can unlock the terminal device. Compared with only voiceprint unlocking or face unlocking, it can be more efficient Ensure the safety of terminal equipment, and improve user experience.

The device for unlocking a terminal device according to the embodiment of the second aspect of the present application includes: a first obtaining module, configured to obtain voiceprint information of the current user when it is detected that the user performs an unlocking operation on the terminal device; a second obtaining module, configured to When it is detected that the user performs an unlocking operation on the terminal device, the three-dimensional face information of the current user is obtained, wherein the three-dimensional face information is obtained by using structured light; A three-dimensional face information database is provided to judge whether the current user is legal; an unlocking module is used to unlock the terminal device if it is legal.

According to the unlocking device of the terminal device according to the embodiment of the present application, when it is detected that the user performs an unlocking operation on the terminal device, the current user's voiceprint information and facial three-dimensional information are obtained, wherein the facial three-dimensional information is obtained by using structured light ; According to the preset voiceprint database and the preset three-dimensional face information database respectively, it is judged whether the current user is legal; if legal, the terminal device is unlocked. The device verifies whether the user is a legitimate user by simultaneously identifying the current user's voiceprint information and facial three-dimensional information. Only legitimate users can unlock the terminal device. Compared with only voiceprint unlocking or face unlocking, it can be more efficient. Ensure the safety of terminal equipment, and improve user experience.

The embodiment of the third aspect of the present application provides one or more non-volatile computer-readable storage media containing computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the processing The device executes the unlocking method of the terminal device according to the embodiment of the first aspect of the present application.

In the mobile terminal according to the embodiment of the fourth aspect of the present application, the mobile terminal includes a memory and a processor, the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processor executes the present invention. The method for unlocking a terminal device according to the embodiment of the first aspect of the application.

According to the mobile terminal of the embodiment of the present application, when it is detected that the user unlocks the terminal device, the current user's voiceprint information and facial three-dimensional information are obtained, wherein the facial three-dimensional information is obtained by using structured light; The preset voiceprint database and the preset three-dimensional face information database are used to judge whether the current user is legal; if legal, the terminal device is unlocked. The mobile terminal verifies whether the user is a legitimate user by simultaneously identifying the current user's voiceprint information and facial three-dimensional information. Only legitimate users can unlock the terminal device. Compared with only voiceprint unlocking or face unlocking, it can Better guarantee the safety of terminal equipment, thereby improving user experience.

The embodiment of the fifth aspect of the present application provides a computer program product, and when the instruction processor in the computer program product executes, executes the terminal device unlocking method of the embodiment of the first aspect of the present application.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein,

FIG. 1 is a flowchart of a method for unlocking a terminal device according to an embodiment of the present application;

FIG. 2 is an exemplary flowchart of obtaining the three-dimensional face information of the current user in the present application;

FIG. 3 is an exemplary flow chart of obtaining the depth image of the current user's head in the present application;

FIG. 4 is an exemplary flow chart of the present application for demodulating the phase information corresponding to each pixel of the structured light image to obtain the depth image of the current user's head depth image;

5(a) to 5(e) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present application;

FIG. 6(a) and FIG. 6(b) are schematic diagrams of scenes of structured light measurement according to an embodiment of the present application;

FIG. 7 is an exemplary flowchart of processing a scene image and a depth image to obtain the 3D face information of the current user in the present application;

Fig. 8 is an exemplary flow chart of judging whether the current user is legal according to the present application;

FIG. 9 is a schematic structural diagram of an unlocking device for a terminal device according to an embodiment of the present application;

Fig. 10 is a schematic diagram of an image processing circuit according to an embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.

The method, device, mobile terminal, and computer-readable storage medium for unlocking a terminal device according to the embodiments of the present application are described below with reference to the accompanying drawings.

Fig. 1 is a flowchart of a method for unlocking a terminal device according to an embodiment of the present application. The method for unlocking a terminal device in this embodiment is applied to a terminal device. Wherein, the terminal device may include a mobile phone, a tablet computer, a smart wearable device, and other hardware devices with various operating systems.

As shown in Figure 1, the unlocking method of the terminal device includes the following steps:

S1. When it is detected that the user performs an unlocking operation on the terminal device, acquire voiceprint information and three-dimensional facial information of the current user, wherein the three-dimensional facial information is acquired by using structured light.

Specifically, voiceprint (Voiceprint) is the sound wave spectrum that carries speech information displayed by electroacoustic instruments. Modern scientific research shows that voiceprint is not only specific, but also relatively stable. Therefore, voiceprint can be regarded as Human identity card. At the same time, because face recognition has the advantages of being natural and not detected by the individual, it can effectively prevent behaviors such as being deceived by camouflage. The terminal device in this embodiment can extract the user's voiceprint information through the configured voice recognition module, and can extract the three-dimensional information of the user's face through the configured imaging device.

For example, when the user lifts up the terminal device or presses the POWER button on the terminal device, a random dynamic password is displayed on the screen of the terminal device in real time. The user's voice corresponding to the random dynamic password, use the configured voice recognition model to extract the user's voiceprint information. At the same time, the imaging device installed on the terminal device first acquires the user's facial image, and then acquires the user's facial three-dimensional information through image processing on the facial image. It should be noted that the manner in which the user unlocks the terminal device is not limited to the illustrated example where the user lifts the terminal device or presses the POWER button on the terminal device.

FIG. 2 is an exemplary flow chart of the present application for acquiring facial 3D information of a current user. As shown in Figure 2, in a possible implementation, the specific implementation of "obtaining the 3D face information of the current user" is as follows:

S10. Acquire a scene image of the head of the current user.

For example, the image capturing device of the terminal device includes a visible light camera 123, and the scene image of the head of the current user is acquired through the visible light camera 123. The visible light camera 123 may be an RGB camera, and the captured image may be a color image.

S11. Project structured light onto the head of the current user to acquire a depth image of the head of the current user.

Fig. 3 is an exemplary flow chart of acquiring the depth image of the current user's head in the present application. As shown in Figure 3, in a possible implementation, the specific implementation of step S11 is:

S110. Project structured light to the head of the current user.

S111. Capture a structured light image modulated by the current user's head.

S112. Demodulate phase information corresponding to each pixel of the structured light image to obtain a depth image of the current user's head.

For example, the imaging device of the terminal device includes a structured light projector. When it is detected that the user unlocks the terminal device, the structured light projector in the terminal device can project structured light to the user's head, and then the terminal device The structured light camera in the device can capture the structured light image modulated by the user's head, and demodulate the phase information corresponding to each pixel of the structured light image to obtain the depth image of the current user's head.

Specifically, after the structured light projector projects a certain pattern of structured light onto the user's head, a structured light image modulated by the user's head will be formed on the surface of the user's head. The structured light camera captures the modulated structured light image, and then demodulates the structured light image to obtain a depth image of the current user's head.

Among them, the pattern of structured light can be laser stripes, gray codes, sinusoidal stripes, non-uniform speckle, etc.

FIG. 4 is an exemplary flow chart of the present application for demodulating phase information corresponding to each pixel of a structured light image to obtain a depth image of a current user's head depth image. As shown in Figure 4, in a possible implementation, the specific implementation of step S112 is:

S1120. Demodulate phase information corresponding to each pixel in the structured light image.

S1121. Convert phase information into depth information.

S1122. Generate a depth image of the current user's head according to the depth information.

Specifically, compared with unmodulated structured light, the phase information of the modulated structured light has changed, and the structured light presented in the structured light image is the structured light after distortion, wherein the changed phase information It can represent the depth information of the object. Therefore, the structured light camera first demodulates the phase information corresponding to each pixel in the structured light image, and then calculates the depth information according to the phase information, thereby obtaining the depth image of the current user's head.

In order to make those skilled in the art more clearly understand the process of collecting the depth images of the first user's face and body according to the structured light, the following uses a widely used grating projection technology (stripe projection technology) as an example to illustrate its details. principle. Among them, grating projection technology belongs to surface structured light in a broad sense.

As shown in Figure 5(a), when using surface structured light projection, firstly generate sinusoidal fringes through computer programming, and project the sinusoidal fringes to the object through the structured light projector, and then use the structured light camera to capture the fringes of the object. The degree of curvature after modulation is then demodulated to obtain the phase, and then the phase is converted into depth information to obtain a depth image. In order to avoid the problem of errors or error coupling, before using structured light for depth information acquisition, it is necessary to calibrate the parameters of the structured light camera and the structured light projector. The calibration includes geometric parameters (for example, the distance between the structured light camera and the structured light projector). Relative position parameters, etc.), the internal parameters of the structured light camera and the internal parameters of the structured light projector, etc.

Specifically, in the first step, the computer is programmed to generate sinusoidal fringes. Since the distorted fringes need to be used to obtain the phase later, for example, the four-step phase shift method is used to obtain the phase, so the four phase differences generated here are stripes, and then the structured light projector time-sharingly projects the four stripes onto the measured object (the mask shown in Figure 5(a)), and the structured light camera captures the picture on the left side of Figure 5(b), and at the same time reads Take the fringes of the reference surface shown on the right side of Fig. 5(b).

The second step is to perform phase recovery. The structured light camera calculates the modulated phase according to the collected four modulated fringe images (ie, the structured light image), and the phase image obtained at this time is a truncated phase image. Because the result obtained by the four-step phase-shift algorithm is calculated by the arctangent function, the modulated phase of the structured light is limited to [-π, π], that is, whenever the modulated phase exceeds [-π , π], which will start all over again. The resulting main value of the phase is shown in Fig. 5(c).

Wherein, during the phase recovery process, transition elimination processing is required, that is, the truncated phase is restored to a continuous phase. As shown in Figure 5(d), the modulated continuous phase map is on the left, and the reference continuous phase map is on the right.

The third step is to subtract the modulated continuous phase from the reference continuous phase to obtain the phase difference (that is, phase information), which represents the depth information of the measured object relative to the reference plane, and then substitute the phase difference into the conversion of phase and depth formula (the parameters involved in the formula have been calibrated), the three-dimensional model of the object to be measured can be obtained as shown in Figure 5(e).

It should be understood that, in practical applications, according to different specific application scenarios, the structured light used in the embodiment of the present application may be in any other pattern besides the above-mentioned grating.

As a possible implementation manner, the present application may also use speckle structured light to collect a depth image of the current user's head.

Specifically, the method for obtaining depth information by speckle structured light is to use a substantially flat diffraction element, which has a relief diffraction structure with a specific phase distribution, and a stepped relief structure with two or more concavo-convex cross sections. The thickness of the substrate in the diffraction element is roughly 1 micron, and the height of each step is not uniform, and the height can range from 0.5 micron to 0.9 micron. The structure shown in Fig. 6(a) is the partial diffraction structure of the collimating beam splitting element of this embodiment. Fig. 6(b) is a cross-sectional side view along section A-A, and the units of the abscissa and ordinate are both micrometers. The speckle pattern generated by speckle structured light is highly random, and the pattern will change with the distance. Therefore, before using speckle structured light to obtain depth information, it is first necessary to calibrate the speckle pattern in space. For example, within the range of 0 to 4 meters from the structured light camera, take a reference plane every 1 cm, then After the calibration is completed, 400 speckle images are saved. The smaller the calibration interval, the higher the accuracy of the obtained depth information. Subsequently, the structured light projector projects the speckle structured light onto the object under test (that is, the first user), and the height difference of the surface of the object under test changes the speckle pattern of the speckle structured light projected on the object under test. After the structured light camera shoots the speckle pattern projected on the object to be measured (i.e. the structured light image), the cross-correlation operation is performed on the speckle pattern and the 400 speckle images saved after previous calibration, and then 400 correlations are obtained. image. The position of the measured object in the space will show a peak on the correlation image, and the depth information of the measured object can be obtained by superimposing the above peaks and interpolating.

Since the ordinary diffraction element diffracts the light beam to obtain most of the diffracted light, but the intensity of each diffracted light varies greatly, and the risk of damage to human eyes is also large. Even if the diffracted light is diffracted twice, the uniformity of the obtained beam is low. Therefore, the projection effect of the light beam diffracted by the common diffraction element on the measured object is relatively poor. In this embodiment, a collimating beam-splitting element is used, which not only has the function of collimating the uncollimated beam, but also has the function of splitting light, that is, the uncollimated light reflected by the mirror passes through the collimating beam-splitting element and then Multiple collimated beams are emitted from different angles, and the cross-sectional areas of the emitted multiple collimated beams are approximately equal, and the energy flux is approximately equal, so that the projection effect of the scattered light after the diffraction of the beam is better. At the same time, the emitted laser light is dispersed to each beam, which further reduces the risk of damage to human eyes. Compared with other evenly arranged structured light, when the speckle structured light achieves the same collection effect, the speckle structured light consumes less lower power.

S12. Process the scene image and the depth image to obtain facial three-dimensional information of the current user.

Since all images are taken of the current user, the scene range of the scene image is basically the same as that of the depth image, and each pixel in the scene image can find the depth information corresponding to the pixel in the depth image.

FIG. 7 is an exemplary flow chart of processing a scene image and a depth image to obtain the 3D face information of the current user in the present application. As shown in Figure 7, in a possible implementation, the specific implementation of step S12 is:

S120, identifying a face area in the scene image;

S121. Obtain depth information corresponding to the face area from the depth image;

S122. Generate facial three-dimensional information of the current user according to the depth information.

The trained deep learning model is used to identify the face area in the scene image, and then the depth information of the face area can be determined according to the corresponding relationship between the scene image and the depth image. Since the face area includes features such as nose, eyes, ears, and lips, the depth data corresponding to each feature in the face area in the depth image is different. For example, when the face is facing the structured light camera, the structure In the depth image captured by the optical camera, the depth data corresponding to the nose may be small, while the depth data corresponding to the ear may be large. Therefore, the above-mentioned depth information of the face area may be a value or a range of values. Wherein, when the depth information of the face area is a value, the value may be obtained by taking an average value of the depth data of the face area; or, may be obtained by taking a median value of the depth data of the face area. For generating corresponding 3D information according to the depth information, reference may be made to the prior art, which will not be repeated here.

S2. Determine whether the current user is legal according to the preset voiceprint database and the preset three-dimensional face information database respectively.

Specifically, the voiceprint information of the legal user is stored in the preset voiceprint library, and if the voiceprint information of the current user has matching voiceprint information in the preset voiceprint library, it means that the voiceprint information of the current user is legal. Voiceprint; the preset facial 3D information database stores the legal user’s facial 3D information. If the current user’s facial 3D information has matching facial 3D information in the preset facial 3D information database, it means The current user's facial 3D information is legitimate facial 3D information. In this embodiment, only when the voiceprint of the current user is determined to be a legitimate voiceprint and the 3D face information of the current user is determined to be legal, the current user is judged to be a legitimate user, thereby avoiding as much as possible A single unlocking method leads to low security of use of the terminal device, and a single unlocking method may unlock the terminal device due to the use of a fake voiceprint or a fake face, which poses a certain security risk.

Further, after step S1 and before step S2, the following steps may also be included:

S0. Determine that the time interval between the first moment of acquiring the voiceprint information and the second moment of acquiring the facial three-dimensional information is within a preset range.

In actual situations, touch terminal devices often occur. For example, the user accidentally touches the POWER power button and speaks a voice. After a long period of time, the user lifts up the terminal device. Therefore, It is necessary to analyze the user touch terminal equipment.

In this embodiment, if it is determined that the time interval between the first moment of acquiring voiceprint information and the second moment of acquiring facial 3D information is within a preset range, it means that the user has the intention to unlock the terminal device; The user accidentally touches the terminal device without intending to unlock the terminal device.

Fig. 8 is an exemplary flow chart of judging whether the current user is legal according to the present application. As shown in Figure 8, in a possible implementation, the specific implementation of step S2 is:

S21. Calculate each first similarity between the voiceprint information of the current user and each preset voiceprint in the preset voiceprint library.

For example, there may be multiple users who have permission to use the terminal device, and multiple preset voiceprints exist in the preset voiceprint library. The voiceprint information of the current user is compared with each preset voiceprint one by one, and the first similarity between the voiceprint information of the current user and the preset voiceprints is calculated. For example, if five preset voiceprints are stored in the preset voiceprint library, five first similarities are calculated.

S22. If the first similarity between the voiceprint information of the current user and any preset voiceprint is greater than a first threshold, determine a first user identifier corresponding to any preset voiceprint.

For example, sort the multiple first similarities obtained in step S21, select the first similarity with the largest similarity and compare it with the first threshold, if it is higher than the first threshold, it means that the voiceprint of the current user is If the legal voiceprint is lower than the first threshold, it means that the current user's voiceprint is an illegal voiceprint. It should be noted that the setting of the first threshold is set according to actual requirements.

It should be pointed out that the preset voiceprint library stores the correspondence between preset voiceprints and user IDs. After determining that the current user's voiceprint is a legitimate voiceprint, determine the The first user ID corresponding to the matched preset voiceprint.

S23. Acquire, according to the first user identifier, facial three-dimensional information corresponding to the first user identifier from the preset human face three-dimensional information database.

In this embodiment, the preset three-dimensional face information database stores the corresponding relationship between the three-dimensional face information and the user identification, and according to the determined first user identification, extracts from the preset three-dimensional information base of the face. Three-dimensional face information corresponding to the user ID.

It should be pointed out that there may be multiple users who have the right to use the terminal device. If both user A and user B have the right to use the terminal equipment, user A faces the imaging device of the terminal equipment, and user B speaks a voice to the random dynamic password displayed on the terminal screen, because user A Both user B and user B are legitimate users, and the terminal device judges the legal voiceprint and legal facial 3D information. At this time, the terminal device will be unlocked successfully.

In this embodiment, after determining that the user's voiceprint is a legal voiceprint, the three-dimensional information of the face corresponding to the user's identification is extracted from the preset face information database according to the user's identification, so as to ensure that only the voiceprint information and face information of the same user Only when the three-dimensional information is successfully matched can the terminal device be unlocked, thereby avoiding the situation where different legal users unlock the terminal device through combination, further ensuring the security of the terminal device and improving user experience.

S24. Determine whether a second similarity between the three-dimensional facial information of the current user and the three-dimensional facial information corresponding to the first user identifier is greater than a second threshold.

Specifically, the facial three-dimensional information of the current user is compared with the facial three-dimensional information corresponding to the first user identifier, and the second similarity between the two is calculated. If the second similarity is higher than the second threshold, It means that the 3D facial information of the current user is legal 3D facial information, and if the second similarity is lower than the second threshold, it means that the 3D facial information of the current user is illegal 3D facial information. It should be noted that the setting of the second threshold is set according to actual requirements.

S3. If legal, unlock the terminal device.

Specifically, if the current user is a legal user, it means that the current user has the authority to use the terminal device, and at this time the terminal device is unlocked for the current user to use; otherwise, if the current user is an illegal user, it means that the current user does not have the authority to use the terminal device. At this point the terminal device remains locked.

The method for unlocking a terminal device provided in this embodiment acquires voiceprint information and three-dimensional facial information of the current user when it is detected that the user performs an unlocking operation on the terminal device, wherein the three-dimensional facial information is obtained by using structured light; According to the preset voiceprint database and the preset three-dimensional face information database respectively, it is judged whether the current user is legal; if legal, the terminal device is unlocked. This method simultaneously identifies the current user's voiceprint information and facial three-dimensional information to verify whether the user is a legitimate user. Only legitimate users can unlock the terminal device. Compared with only voiceprint unlocking or face unlocking, it can be more efficient Ensure the safety of terminal equipment, and improve user experience.

In order to implement the foregoing embodiments, the present application further proposes an unlocking apparatus for a terminal device according to the embodiments of the present application.

Fig. 9 is a schematic structural diagram of an unlocking device for a terminal device according to an embodiment of the present application.

As shown in FIG. 9 , the unlocking apparatus for a terminal device in this embodiment of the present application may include a first acquiring module, a second acquiring module, a judging module, and an unlocking module.

Wherein, the first acquiring module is configured to acquire the voiceprint information of the current user when it is detected that the user unlocks the terminal device.

Wherein, the second acquisition module is configured to acquire the current user's facial three-dimensional information when it is detected that the user performs an unlocking operation on the terminal device, wherein the facial three-dimensional information is acquired by using structured light;

Wherein, the judging module is used to judge whether the current user is legal according to the preset voiceprint database and the preset three-dimensional face information database respectively.

Wherein, the unlocking module is configured to unlock the terminal device if it is legal.

Further, the device further includes: a determining module, used for determining whether the current user is legal or not, before the judging module determines whether the first moment of acquiring the voiceprint information and the second moment of acquiring the facial three-dimensional information The time interval between moments is within a preset range.

Further, the judging module is specifically used for:

respectively calculating the first similarities between the voiceprint information of the current user and each preset voiceprint in the preset voiceprint library;

If the first similarity between the voiceprint information of the current user and any preset voiceprint is greater than a first threshold, then determine the first user identifier corresponding to any preset voiceprint;

According to the first user identifier, acquire facial three-dimensional information corresponding to the first user identifier from the preset facial three-dimensional information database;

It is judged whether the second similarity between the facial three-dimensional information of the current user and the facial three-dimensional information corresponding to the first user identifier is greater than a second threshold.

Further, the second acquisition module includes: a first image acquisition unit, a second image acquisition unit, and a processing unit;

The first image acquisition unit is configured to project visible light onto the head of the current user to acquire a scene image of the head of the current user;

The second image acquisition unit is configured to project structured light onto the head of the current user to acquire a depth image of the head of the current user;

The processing unit is configured to process the scene image and the depth image to acquire the 3D face information of the current user.

Further, the processing unit is specifically used for:

identifying face regions in the image of the scene;

acquiring depth information corresponding to the face region from the depth image;

Generate facial three-dimensional information of the current user according to the depth information.

Wherein, it should be noted that the foregoing explanations on the embodiment of the terminal device unlocking method are also applicable to the device for unlocking the terminal device in this embodiment, and its implementation principles are similar, so details are not repeated here.

The device for unlocking a terminal device provided in this embodiment acquires voiceprint information and three-dimensional facial information of the current user when detecting that the user performs an unlocking operation on the terminal device, wherein the three-dimensional facial information is obtained by using structured light; According to the preset voiceprint database and the preset three-dimensional face information database respectively, it is judged whether the current user is legal; if legal, the terminal device is unlocked. The device verifies whether the user is a legitimate user by simultaneously identifying the current user's voiceprint information and facial three-dimensional information. Only legitimate users can unlock the terminal device. Compared with only voiceprint unlocking or face unlocking, it can be more efficient. Ensure the safety of terminal equipment, and improve user experience.

In order to implement the above embodiments, the present application also proposes a mobile terminal.

A mobile terminal includes the device for unlocking a terminal device according to the embodiment of the second aspect of the present application.

According to the mobile terminal of the embodiment of the present application, when it is detected that the user unlocks the terminal device, the current user's voiceprint information and facial three-dimensional information are obtained, wherein the facial three-dimensional information is obtained by using structured light; The preset voiceprint database and the preset three-dimensional face information database are used to judge whether the current user is legal; if legal, the terminal device is unlocked. By simultaneously identifying the current user's voiceprint information and facial 3D information to verify whether the user is a legitimate user, only legitimate users can unlock the terminal device, which is better than unlocking only voiceprint or face unlocking. Ensure the safety of terminal equipment, thereby improving user experience.

The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, The processor is made to execute the aforementioned method for unlocking the terminal device.

In order to implement the above embodiments, the present application also proposes a mobile terminal.

The above-mentioned mobile terminal includes an image processing circuit, which may be implemented by hardware and/or software components, and may include various processing units defining an ISP (Image Signal Processing, image signal processing) pipeline. Fig. 10 is a schematic diagram of an image processing circuit according to an embodiment of the present application. As shown in FIG. 10 , for ease of description, only various aspects of the image processing technology related to the embodiment of the present application are shown.

As shown in FIG. 10 , the image processing circuit of the mobile terminal 1200 includes an imaging device 10 , an ISP processor 30 and a control logic 40 . The imaging device 10 may include a visible light camera 123 , a structured light projector 121 , and a structured light camera 122 .

The imaging device 10 includes a visible light camera 11 and a depth image acquisition component 12 .

Specifically, the visible light camera 123 includes an image sensor 1231 and a lens 1232. The visible light camera 123 can be used to capture the color information of the current user to obtain a scene image, wherein the image sensor 1231 includes a color filter array (such as a Bayer filter array), and the lens 1232 The number of can be one or more. When the visible light camera 123 acquires the scene image, each imaging pixel in the image sensor 1231 senses the light intensity and wavelength information from the shooting scene to generate a set of original image data; the image sensor 1231 sends the set of original image data to the ISP In the processor 30, the ISP processor 30 performs operations such as denoising and interpolation on the original image data to obtain a color scene image. The ISP processor 30 can process each image pixel in the raw image data one by one in a variety of formats, for example, each image pixel can have a bit depth of 8, 10, 12 or 14 bits, and the ISP processor 30 can process each image pixel in the same or Different bit depths are processed for each image pixel.

Specifically, the structured light projector 121 projects the structured light onto the current user's head. Wherein, the structured light pattern may be laser stripes, gray codes, sinusoidal stripes, or randomly arranged speckle patterns and the like. The structured light camera 122 may include an image sensor 1221 and a lens 1222 . Wherein, the number of lenses 1222 may be one or more. The image sensor 1221 is used to capture the structured light image projected by the structured light projector 121 onto the head of the current user. The structured light image can be sent by the image acquisition module 120 to the ISP processor 30 for demodulation, phase recovery, phase information calculation and other processing to obtain the depth information of the first user.

Wherein, the above-mentioned image sensor 1221 is also used to capture the structured light image projected by the structured light projector 121 onto the head of the current user, and send the structured light image to the ISP processor 30, and the ISP processor 30 decodes the structured light image. Call to obtain the depth information of the measured object (the measured object refers to the current user's head in this embodiment). At the same time, the image sensor 1221 can also capture the color information of the measured object. Of course, the two image sensors 1221 can also capture the structured light image and color information of the object under test respectively.

Wherein, taking speckle structured light as an example, the ISP processor 30 demodulates the structured light image, which specifically includes collecting the speckle image of the object under test from the structured light image, and combining the speckle image of the object under test with the reference The image data of the speckle image is calculated according to a predetermined algorithm, and the moving distance of each speckle point in the speckle image on the object under test relative to the reference speckle point in the reference speckle image is obtained. The depth value of each speckle point of the speckle image is converted and calculated by using the triangulation method, and the depth information of the measured object is obtained according to the depth value.

Of course, the depth image information can also be obtained through binocular vision or a method based on time-of-flight TOF, and there is no limitation here, as long as the method that can obtain or calculate the depth information of the measured object belongs to this implementation The range covered by the method.

After the ISP processor 30 receives the color information of the object under test captured by the image sensor 1221 , it can process the image data corresponding to the color information of the object under test. ISP processor 30 analyzes the image data to obtain image statistics that may be used to determine and/or control one or more parameters of imaging device 10 . The image sensor 1221 can include a color filter array (such as a Bayer filter), and the image sensor 1221 can obtain light intensity and wavelength information captured by each imaging pixel of the image sensor 1221, and provide a set of raw images that can be processed by the ISP processor 30. image data.

The ISP processor 30 processes raw image data on a pixel-by-pixel basis in various formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and ISP processor 30 may perform one or more image processing operations on raw image data, gather image statistics about the image data. Among other things, image processing operations can be performed with the same or different bit depth precision.

ISP processor 30 may also receive pixel data from image sensor 1221 . The image sensor 1221 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include a DMA (Direct Memory Access, Direct Memory Access) feature.

When raw image data is received, ISP processor 30 may perform one or more image processing operations.

After the ISP processor 30 acquires the color information and depth information of the measured object, it can be fused to obtain a three-dimensional image. Wherein, the feature of the corresponding object under test may be extracted by at least one of an appearance contour extraction method or a contour feature extraction method. For example, the features of the measured object are extracted by active shape modeling method ASM, active appearance modeling method AAM, principal component analysis method PCA, discrete cosine transform method DCT and other methods, which are not limited here. Then, the features of the measured object extracted from the depth information and the features of the measured object extracted from the color information are subjected to registration and feature fusion processing. The fusion processing referred to here can be to directly combine the features extracted from the depth information and color information, or to combine the same features in different images after setting weights, or there can be other fusion methods, and finally according to the fused features to generate a 3D image.

The image data for the three-dimensional image may be sent to image memory 20 for additional processing before being displayed. The ISP processor 30 receives the processed data from the image memory 20, and performs image data processing in the raw domain and in the RGB and YCbCr color spaces on the processed data. The image data of the three-dimensional image can be output to the display 60 for viewing by the user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit, graphics processor). In addition, the output of the ISP processor 30 can also be sent to the image memory 20 , and the display 60 can read image data from the image memory 20 . In one embodiment, image memory 20 may be configured to implement one or more frame buffers. In addition, the output of the ISP processor 30 may be sent to an encoder/decoder 50 for encoding/decoding image data. The encoded image data may be saved and decompressed prior to display on the display 60 device. Encoder/decoder 50 may be implemented by a CPU or GPU or a coprocessor.

The image statistics determined by the ISP processor 30 may be sent to the control logic 40 unit. Control logic 40 may include a processor and/or microcontroller executing one or more routines (eg, firmware) that may determine control parameters for imaging device 10 based on received image statistics.

The following are the steps of using the image processing technology in Figure 10 to realize the unlocking method of the terminal device:

S1', when it is detected that the user unlocks the terminal device, acquire voiceprint information and three-dimensional facial information of the current user, wherein the three-dimensional facial information is acquired by using structured light;

S2', judging whether the current user is legal according to the preset voiceprint database and the preset three-dimensional face information database respectively;

S3', if legal, unlock the terminal device.

Wherein, it should be noted that the foregoing explanations of the embodiment of the method for unlocking a terminal device are also applicable to the mobile terminal of this embodiment, and its implementation principles are similar, so details will not be repeated here.

A computer program product, when an instruction processor in the computer program product executes, executes the aforementioned terminal device unlocking method.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment for use. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as it may be possible, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable means if necessary. Processing to obtain programs electronically and store them in computer memory.

It should be understood that each part of the present application may be realized by hardware, software, firmware or a combination thereof. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGAs), Field Programmable Gate Arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed , including one or a combination of the steps of the method embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (12)

1. A method for unlocking a terminal device, comprising: When it is detected that the user performs an unlocking operation on the terminal device, the current user's voiceprint information and facial three-dimensional information are acquired, wherein the facial three-dimensional information is acquired by using structured light; Judging whether the current user is legal according to the preset voiceprint database and the preset three-dimensional face information database respectively; If legal, unlock the terminal device. 2. The method according to claim 1, characterized in that before the judging whether the current user is legal, comprises: It is determined that the time interval between the first moment of acquiring the voiceprint information and the second moment of acquiring the facial three-dimensional information is within a preset range. 3. The method according to claim 1, wherein the judging whether the current user is legal comprises: respectively calculating the first similarities between the voiceprint information of the current user and each preset voiceprint in the preset voiceprint library; If the first similarity between the voiceprint information of the current user and any preset voiceprint is greater than a first threshold, then determine the first user identifier corresponding to any preset voiceprint; According to the first user identifier, acquire facial three-dimensional information corresponding to the first user identifier from the preset facial three-dimensional information database; It is judged whether the second similarity between the facial three-dimensional information of the current user and the facial three-dimensional information corresponding to the first user identifier is greater than a second threshold. 4. The method according to claim 1, wherein said obtaining the facial three-dimensional information of the current user comprises: projecting visible light onto the current user's head to acquire a scene image of the current user's head; projecting structured light onto the head of the current user to obtain a depth image of the head of the current user; The scene image and the depth image are processed to obtain the 3D face information of the current user. 5. The method according to claim 4, wherein the processing the scene image and the depth image to obtain the current user's facial three-dimensional information comprises: identifying face regions in the image of the scene; acquiring depth information corresponding to the face region from the depth image; Generate facial three-dimensional information of the current user according to the depth information. 6. An unlocking device for terminal equipment, characterized in that it comprises: The first obtaining module is used to obtain the voiceprint information of the current user when it is detected that the user performs an unlocking operation on the terminal device; The second acquiring module is configured to acquire the 3D facial information of the current user when it is detected that the user performs an unlocking operation on the terminal device, wherein the 3D facial information is acquired by using structured light; The judging module is used to judge whether the current user is legal according to the preset voiceprint database and the preset three-dimensional face information database respectively; The unlocking module is configured to unlock the terminal device if it is legal. 7. The device according to claim 6, further comprising: a determining module, used for determining whether the first moment of obtaining the voiceprint information is related to the obtained time before the judging module judges whether the current user is legal. The time interval between the second moments of the facial three-dimensional information is within a preset range. 8. The device according to claim 6, wherein the judging module is specifically used for: respectively calculating the first similarities between the voiceprint information of the current user and each preset voiceprint in the preset voiceprint library; If the first similarity between the voiceprint information of the current user and any preset voiceprint is greater than a first threshold, then determine the first user identifier corresponding to any preset voiceprint; According to the first user identifier, acquire facial three-dimensional information corresponding to the first user identifier from the preset facial three-dimensional information database; It is judged whether the second similarity between the facial three-dimensional information of the current user and the facial three-dimensional information corresponding to the first user identifier is greater than a second threshold. 9. The device according to claim 6, wherein the second acquisition module comprises: a first image acquisition unit, a second image acquisition unit, and a processing unit; The first image acquisition unit is configured to project visible light onto the head of the current user to acquire a scene image of the head of the current user; The second image acquisition unit is configured to project structured light onto the head of the current user to acquire a depth image of the head of the current user; The processing unit is configured to process the scene image and the depth image to acquire the 3D face information of the current user. 10. The device according to claim 9, wherein the processing unit is specifically configured to: identifying face regions in the image of the scene; acquiring depth information corresponding to the face region from the depth image; Generate facial three-dimensional information of the current user according to the depth information. 11. One or more non-transitory computer-readable storage media containing computer-executable instructions which, when executed by one or more processors, cause the processors to perform The unlocking method of the terminal device described in any one of 5. 12. A mobile terminal, comprising a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes any one of claims 1 to 5. The unlocking method of the terminal equipment described in item.