WO2024146165A1 - Human eye positioning method and apparatus, and computing device and storage medium - Google Patents

Abstract

The present application relates to a human eye positioning method and apparatus, and a device and a medium. The method comprises: performing facial key point detection on an image to be processed that comprises a human face, so as to obtain a facial key point in said image; on the basis of the facial key point in said image, determining head posture information for indicating a deflection angle between the head of a person and a preset direction; and determining three-dimensional human eye coordinates on the basis of a pupil distance between human eyes in said image, camera parameters of a camera for capturing said image, the head posture information, and pixel coordinates of the left eye and pixel coordinates of the right eye in said image.

Description

Human eye positioning method, device, computing device and storage medium Technical Field

The present application relates to the technical field of computer vision and image processing, and in particular to a method, apparatus, computing device and storage medium for locating human eyes.

Background technique

Compared with binocular cameras and depth cameras, monocular cameras have the advantages of simple structure and high cost performance, and have been widely used in human eye positioning and ranging.

In the related technology, when locating and measuring the distance between the human eye through a monocular camera, the pixel coordinates of the left and right eyes in the image taken by the camera are directly used as the eye coordinates to achieve eye positioning, and then the principle of near is larger and far is smaller is used. Through the actual pupil length of the human eye and the pixel distance between the left and right eyes in the image taken by the camera, the distance between the face and the camera is estimated according to the principle of similar triangle projection to achieve distance measurement.

However, if the face is not facing the camera, the pixel distance between the two eyes in the image taken by the camera will become smaller due to the side face, which will cause a large error in the eye coordinates. At this time, if the distance between the face and the camera is simply estimated based on the coordinates of the two eyes in the image, the estimated distance between the face and the camera will have a large error, resulting in poor accuracy in the distance measurement process.

Summary of the invention

The present application provides a human eye positioning method, apparatus, computing device and storage medium to address deficiencies in the related art.

According to a first aspect of an embodiment of the present application, a method for locating human eyes is provided, the method comprising: performing facial key point detection on an image to be processed that includes a human face, and obtaining the facial key points in the image to be processed; determining head posture information based on the facial key points in the image to be processed, the head posture information being used to indicate a deflection angle between a person's head and a preset direction; determining the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of a camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed.

In some embodiments of the present application, head posture information is determined based on facial key points in an image to be processed, including: determining a head posture matrix based on facial key points in an image to be processed, the head posture matrix being used to represent the posture of the head in the image to be processed in the form of a matrix; determining values of target parameters based on the head posture matrix; determining head posture information based on the values of the target parameters and the head posture matrix.

In some embodiments of the present application, the head posture information includes a pitch angle, a rotational yaw angle, and a rotational roll angle;

Based on the value of the target parameter and the head posture matrix, the head posture information is determined, including: when the value of the target parameter is greater than or equal to the set threshold, the pitch angle is determined based on the elements at the first set position and the second set position in the head posture matrix, the rotation yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the rotation roll angle is determined based on the elements at the fourth set position and the fifth set position in the head posture matrix; when the value of the target parameter is less than the set threshold, the pitch angle is determined based on the elements at the sixth set position and the seventh set position in the head posture matrix, the rotation yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the value of the rotation roll angle is determined as the set angle value.

In some embodiments of the present application, the pixel coordinates are two-dimensional coordinates including abscissas and ordinates, and the three-dimensional coordinates of the human eye include abscissas, ordinates, and depth coordinates of the left eye and abscissas, ordinates, and depth coordinates of the right eye;

Based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the three-dimensional coordinates of the human eye are determined, including: based on the pupil distance of the human eye, the camera parameters, the head posture information, the pixel coordinates of the left eye and the pixel coordinates of the right eye, the depth coordinates of the center points of the two eyes are determined; based on the pupil distance of the human eye, the head posture matrix, the depth coordinates of the center points of the two eyes, respectively Determine the depth coordinates of the left eye and the depth coordinates of the right eye; based on the camera parameters, the pixel coordinates of the left eye, the pixel coordinates of the right eye, the depth coordinates of the left eye and the depth coordinates of the right eye, determine the horizontal coordinate and vertical coordinate of the left eye, and the horizontal coordinate and vertical coordinate of the right eye.

In some embodiments of the present application, after determining the head posture matrix based on the facial key points in the image to be processed, the method also includes: determining the three-dimensional coordinates of the center points of the two eyes based on the camera parameters and the pixel coordinates of the left eye and the right eye in the image to be processed; determining the rotation matrix based on the three-dimensional coordinates of the center points of the two eyes, and the rotation matrix is used to indicate the mapping relationship between the local coordinate system and the global coordinate system; based on the rotation matrix and the head posture matrix, determining the target head posture matrix corresponding to the head posture matrix in the global coordinate system.

In some embodiments of the present application, the three-dimensional coordinates of the human eye are determined based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, including: determining the target head posture information based on the target head posture matrix; determining the three-dimensional coordinates of the human eye based on the pupil distance of the human eye, the camera parameters, the target head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye.

In some embodiments of the present application, facial key point detection is performed on an image to be processed that includes a face to obtain facial key points in the image to be processed, including: performing face detection based on the image to be processed to determine a face positioning frame in the image to be processed; performing face key point detection based on the image portion corresponding to the face positioning frame to obtain facial key points in the image to be processed.

In some embodiments of the present application, after determining the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the method also includes: determining the distance between the human eye and the camera based on the three-dimensional coordinates of the human eye; when the distance is less than the set distance threshold, sending a prompt message, the prompt message being used to prompt the user to increase the distance between the human eye and the camera.

In some embodiments of the present application, before performing facial key point detection on the image to be processed including the face, the quality of the face in the image to be processed is also detected to filter out faces with poor image quality.

According to a second aspect of an embodiment of the present application, a human eye positioning device is provided, the device comprising: a detection module, used to detect facial key points of an image to be processed including a human face, and obtain the facial key points in the image to be processed; a first determination module, used to determine head posture information based on the facial key points in the image to be processed, the head posture information being used to indicate a deflection angle between a person's head and a preset direction; a second determination module, used to determine the three-dimensional coordinates of the human eye based on the human eye pupil distance in the image to be processed, the camera parameters of a camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed.

In some embodiments of the present application, the first determination module, when used to determine the head posture information based on the facial key points in the image to be processed, is used to: determine the head posture matrix based on the facial key points in the image to be processed, the head posture matrix being used to represent the posture of the head in the image to be processed in the form of a matrix; determine the value of the target parameter based on the head posture matrix; determine the head posture information based on the value of the target parameter and the head posture matrix.

In some embodiments of the present application, the head posture information includes a pitch angle, a rotational yaw angle, and a rotational roll angle;

The first determination module, when used to determine the head posture information based on the value of the target parameter and the head posture matrix, is used to: when the value of the target parameter is greater than or equal to a set threshold, determine the pitch angle based on the elements at the first set position and the second set position in the head posture matrix, determine the rotation yaw angle based on the elements at the third set position in the head posture matrix and the value of the target parameter, and determine the rotation roll angle based on the elements at the fourth set position and the fifth set position in the head posture matrix; when the value of the target parameter is less than the set threshold, determine the pitch angle based on the elements at the sixth set position and the seventh set position in the head posture matrix, determine the rotation yaw angle based on the elements at the third set position in the head posture matrix and the value of the target parameter, and determine the value of the rotation roll angle as the set angle value.

In some embodiments of the present application, the pixel coordinates are two-dimensional coordinates including abscissas and ordinates, and the three-dimensional coordinates of the human eye include abscissas, ordinates, and depth coordinates of the left eye and abscissas, ordinates, and depth coordinates of the right eye; the second determination module is used to determine the image based on the pupil distance of the human eye in the image to be processed, the camera used to shoot the image to be processed, When determining the three-dimensional coordinates of the human eye based on the camera parameters, head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, it is used to: determine the depth coordinates of the center points of the two eyes based on the pupil distance of the human eye, the camera parameters, the head posture information, the pixel coordinates of the left eye and the pixel coordinates of the right eye; determine the depth coordinates of the left eye and the depth coordinates of the right eye respectively based on the pupil distance of the human eye, the head posture matrix, and the depth coordinates of the center points of the two eyes; determine the horizontal and vertical coordinates of the left eye, as well as the horizontal and vertical coordinates of the right eye based on the camera parameters, the pixel coordinates of the left eye, the pixel coordinates of the right eye, the depth coordinates of the left eye, and the depth coordinates of the right eye.

In some embodiments of the present application, the device also includes: a third determination module, used to determine the three-dimensional coordinates of the center points of the two eyes based on camera parameters, the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed; the third determination module is also used to determine the rotation matrix based on the three-dimensional coordinates of the center points of the two eyes, and the rotation matrix is used to indicate the mapping relationship between the local coordinate system and the global coordinate system; the third determination module is also used to determine the target head posture matrix corresponding to the head posture matrix in the global coordinate system based on the rotation matrix and the head posture matrix.

In some embodiments of the present application, the second determination module, when used to determine the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, is used to: determine the target head posture information based on the target head posture matrix; determine the three-dimensional coordinates of the human eye based on the pupil distance of the human eye, the camera parameters, the target head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye.

In some embodiments of the present application, the detection module, when used to perform facial key point detection on an image to be processed that includes a face and obtain the facial key points in the image to be processed, is used to: perform face detection based on the image to be processed to determine a face positioning frame in the image to be processed; perform facial key point detection based on the image portion corresponding to the face positioning frame to obtain the facial key points in the image to be processed.

In some embodiments of the present application, the device also includes: a fourth determination module, used to determine the distance between the human eye and the camera based on the three-dimensional coordinates of the human eye; a sending module, used to send a prompt message when the distance is less than a set distance threshold, and the prompt message is used to prompt the user to increase the distance between the human eye and the camera.

According to a third aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the operations performed by the human eye positioning method provided in the first aspect and any one of the embodiments of the first aspect are implemented.

According to a fourth aspect of an embodiment of the present application, a non-volatile computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the operations performed by the human eye positioning method provided in the first aspect and any one of the embodiments of the first aspect are implemented.

According to a fifth aspect of the embodiments of the present application, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, it implements the operations performed by the human eye positioning method provided in the first aspect and any one of the embodiments of the first aspect.

According to the above embodiments, the present application performs facial key point detection on an image to be processed that includes a human face to obtain the facial key points in the image to be processed; thereby, based on the facial key points in the image to be processed, head posture information for indicating the deflection angle between a person's head and a preset direction is determined; and then based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the three-dimensional coordinates of the human eye are determined, so as to achieve the purpose of correcting the human eye coordinates according to the head posture information, so that the position of the human eye in the image can be determined more accurately.

It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

FIG1 is a schematic diagram of an application scenario of a method for locating human eyes according to an embodiment of the present application.

FIG. 2 is a flow chart of a method for locating human eyes according to an embodiment of the present application.

FIG. 3 is a schematic diagram of key points of a human face according to an embodiment of the present application.

FIG. 4 is a schematic diagram showing a pitch angle, a rotational yaw angle, and a rotational roll angle according to an embodiment of the present application.

FIG. 5 is a flow chart of a method for locating human eyes according to an embodiment of the present application.

FIG. 6 is a schematic structural diagram of a human eye positioning device according to an embodiment of the present application.

FIG. 7 is a schematic diagram of the structure of a computing device according to an embodiment of the present application.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

The present application provides a method for locating human eyes, which is used to correct the coordinates of the human eyes in the image to be processed according to the head posture information of the human head in the image to be processed after acquiring the image to be processed taken by a camera, so as to determine more accurate three-dimensional coordinates of the human eyes in the image to be processed, thereby improving the accuracy of the determined human eye position.

In a possible application scenario, the eye positioning method provided by the present application can be applied to a distance measurement scenario to determine the distance between the human eye and the display device. Optionally, the display device can be a desktop computer, a portable computer, a smart phone, a tablet computer, a laptop computer, etc. A camera can be integrated in the display device to capture the image to be processed through the camera, so that the eye positioning method provided by the present application can determine the more accurate three-dimensional coordinates of the human eye in the image to be processed, and then determine the distance between the human eye and the camera based on the determined three-dimensional coordinates of the human eye. The distance between the human eye and the camera is also the distance between the human eye and the display device, so that when the distance between the human eye and the display device is too close, a prompt message can be issued through the display device to prompt the user to keep a distance from the display device, helping to prevent myopia in the user.

In another possible application scenario, the eye positioning method provided by the present application can be applied to a naked-eye three-dimensional (3D) scene. For example, the image to be processed can be photographed by a camera integrated in a naked-eye 3D device, so that the eye positioning method provided by the present application can determine a more accurate three-dimensional coordinate of the eye in the image to be processed, so that the image to be displayed in front of the left eye and the right eye can be determined according to the determined three-dimensional coordinates of the eye (including the left eye coordinates and the right eye coordinates), so that the user can enjoy a three-dimensional image that is as real as a real object without wearing external tools such as polarized glasses (such as 3D glasses). Among them, the naked-eye 3D device can be a naked-eye 3D advertising machine, a naked-eye 3D notebook, etc., and the present application does not limit the device type of the naked-eye 3D device.

Taking the application of the human eye positioning method provided by the present application in a naked-eye 3D scene as an example, refer to Figure 1, which is a schematic diagram of an application scenario of a human eye positioning method according to an embodiment of the present application. As shown in Figure 1, after the naked-eye 3D device captures an image through a camera (such as a monocular camera), the human eye positioning method provided by the present application can be used to determine the three-dimensional coordinates of the human eye in the image, thereby rearranging the 3D display pixels of each frame image in the 3D video source based on the three-dimensional coordinates of the human eye to obtain multiple naked-eye 3D synthetic images. The naked-eye 3D device can then display the naked-eye 3D synthetic images so that viewers can see high-quality 3D effects at different positions within a wide range.

It should be noted that, no matter which of the above scenarios, the camera can be a monocular camera (such as a monocular RGB camera) or an infrared grayscale image camera. In addition, in addition to shooting an image to be processed to determine the three-dimensional coordinates of the human eye based on the image to be processed, the camera can also shoot a video to use at least one video frame in the video as the image to be processed to achieve the determination of the three-dimensional coordinates of the human eye. This application does not limit the specific type of the image to be processed.

The above is only an exemplary description of the application scenario of the present application and does not constitute a limitation on the application scenario of the present application. In more possible implementations, the human eye positioning method provided by the present application can also be applied in other scenarios, and the present application does not limit the specific application scenario.

The above-mentioned human eye positioning method can be executed by a computing device, and the computing device can be the display device or the naked eye 3D device mentioned in the above-mentioned exemplary scenario, or the computing device can also be connected to the display device or the naked eye 3D Other devices with which the device communicates, such as terminal devices or servers. The terminal device may be a desktop computer, a portable computer, a notebook computer, a smart phone, a tablet computer, etc. The server may be one server, multiple servers, a server cluster, a cloud computing platform, etc. This application does not limit the device type of the computing device.

In the case where the computing device is the display device or naked-eye 3D device mentioned in the above exemplary scenario, the computing device itself can capture the image to be processed, so as to determine the three-dimensional coordinates of the human eye based on the image to be processed by the human eye positioning method provided in this application. In the case where the computing device is other devices that can communicate with the display device or the naked-eye 3D device, after the display device or the naked-eye 3D device captures the image to be processed, it can send the image to the computing device, so that the computing device can determine the three-dimensional coordinates of the human eye based on the received image to be processed by the human eye positioning method provided in this application, and then send the determined three-dimensional coordinates of the human eye to the display device or the naked-eye 3D device, so that the display device or the naked-eye 3D device can obtain more accurate three-dimensional coordinates of the human eye.

After introducing the application scenarios and execution entities of the present application, the human eye positioning method provided by the present application is described in detail below.

Referring to FIG. 2 , FIG. 2 is a flow chart of a method for locating human eyes according to an embodiment of the present application. As shown in FIG. 2 , the method includes:

Step 201: perform facial key point detection on an image to be processed that includes a face, and obtain facial key points in the image to be processed.

Generally speaking, there are five most critical points for marking a face, namely, the left and right corners of the mouth, the centers of the two eyes (that is, the center of the left eye and the center of the right eye), and the tip of the nose. These five key points belong to the key points inside the face, and the posture of the face can be determined based on these five key points. Referring to FIG. 3 , FIG. 3 is a schematic diagram of a face key point according to an embodiment of the present application. As shown in FIG. 3 , the face key points determined by step 201 include the five key points of the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

Step 202: determine head posture information based on the facial key points in the image to be processed, where the head posture information is used to indicate a deflection angle between the head of the person and a preset direction.

It should be noted that when the human face is facing the camera, the pixel coordinates of the human eye in the captured image and the position of the human eye on the face are relatively accurate. Once the human face in the captured image is sideways, the pixel coordinates of the human eye in the captured image and the position of the human eye on the face will be quite different. At this time, the head posture information is determined through step 202. The head posture information can indicate the deflection angle between the face and the direction facing the camera (that is, the preset direction), so that the pixel coordinates of the human eye can be corrected based on the head posture information, and the depth coordinates of the human eye can be determined to achieve the purpose of accurately locating the three-dimensional coordinates of the human eye.

Step 203 : determining the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed.

Among them, the pupil distance of the human eye in the image to be processed can be the pixel distance between the left and right eyes in the image to be processed. Optionally, the pixel distance between the left and right eyes in the image to be processed can be determined based on the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed to obtain the pupil distance of the human eye in the image to be processed.

It should be noted that the horizontal and vertical coordinates in the determined three-dimensional coordinates of the human eye are obtained by correcting the pixel coordinates of the human eye in the processed image based on the head posture information, and the depth coordinates are also expanded compared to the pixel coordinates, so that the determined human eye coordinates are more accurate three-dimensional coordinates.

The present application performs facial key point detection on an image to be processed that includes a human face to obtain the facial key points in the image to be processed; thereby, based on the facial key points in the image to be processed, head posture information indicating the deflection angle between a person's head and a preset direction is determined; and then based on the pupil distance of the human eye in the image to be processed, the camera parameters of a camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the three-dimensional coordinates of the human eye are determined, so as to achieve the purpose of correcting the coordinates of the human eye according to the head posture information, thereby more accurately determining the position of the human eye in the image.

After introducing the basic implementation process of the human eye positioning method provided by this application, the following describes the human eye positioning method. Various alternative embodiments of the method.

In some embodiments, for step 201, when performing facial key point detection on the image to be processed including the face to obtain the facial key points in the image to be processed, it can be implemented by the following steps:

Step 2011: Perform face detection based on the image to be processed to determine a face positioning frame in the image to be processed.

In a possible implementation, the image to be processed may be input into a face detection model, and the face detection model may be used to process the image to be processed so as to output a face location frame in the image to be processed.

Among them, the face detection model can detect the face target in the image to be processed, and the position of the detected face can be marked with a face positioning frame to determine the face positioning frame in the image to be processed.

It should be noted that the face positioning frame can be a rectangular frame. In the case where the face positioning frame is a rectangular frame, the diagonal vertices of the rectangular frame can be marked to achieve the annotation of the face positioning frame. Optionally, the vertex located at the upper left corner of the rectangular frame and the vertex located at the lower right corner of the rectangular frame can be marked. According to the two marked vertices, the position and size of the rectangular frame can be determined. For example, the vertex at the upper left corner of the rectangular frame can be marked as (x1, y1), and the vertex at the lower right corner of the rectangular frame can be recorded as (x2, y2).

Optionally, the face detection model may be a face detector based on an adaptive boosting tree (Adboost), or the face detection model may be a deep learning face detection model. Taking the face detection model as a deep learning face detection model as an example, a Retinaface model or a Yolov7face model may be used as a face detection model to implement face detection in the image to be processed.

Step 2012: perform facial key point detection based on the image portion corresponding to the face positioning frame to obtain facial key points in the image to be processed.

In a possible implementation, a facial key point detection model may be used to perform facial key point detection based on the image portion corresponding to the face positioning frame to obtain facial key points in the image to be processed.

Optionally, after determining the face positioning frame in the image to be processed through step 2011, the image to be processed can be cropped based on the face positioning frame to obtain a face image that only includes the image portion corresponding to the face positioning frame, thereby performing face key point detection based on the face image to obtain the face key points in the image to be processed.

That is, the face image can be input into the face key point detection model, and the face key point detection model can be used to perform face key point detection based on the face image to realize the detection of face key points in the face image, thereby obtaining the face key points in the image to be processed.

Among them, the facial key point detection model can be a deep learning model. For example, the facial key point detection model can be a convolutional neural network (CNN) model, such as a CNN model based on a residual network (Residual Network, ResNet). Optionally, the facial key point detection model can also be other types of models, which is not limited in this application.

Optionally, before performing facial key point detection based on the image portion corresponding to the face positioning frame, the quality of the face in the image to be processed may also be detected to filter out faces with poor image quality.

In a possible implementation, a face quality detection model may be used to perform face quality detection on an image portion corresponding to a face positioning frame in the image to be processed, so as to filter out faces whose image quality does not meet the requirements in the image to be processed.

Among them, the face quality detection model can be a CNN model, such as a CNN model based on ResNet. Optionally, the face quality detection model can also be other types of models, which is not limited in this application. It should be noted that the face key point detection model and the face quality detection model can be the same model, that is, the same model can provide the functions of face key point detection and face quality detection at the same time, or the face key point detection model and the face quality detection model can be two models, so that face key point detection and face quality detection can be realized respectively through these two models.

By performing face quality detection on the image portion corresponding to the face positioning frame in the image to be processed, faces whose image quality does not meet the requirements in the image to be processed can be filtered out, so that only face key point detection needs to be performed on faces whose image quality meets the requirements in the subsequent processing, thereby reducing the processing pressure of the computing device.

After determining the key points of the face through the above process, step 202 can be performed based on the key points of the face in the image to be processed. Facial key points, determine the head posture information.

In some embodiments, for step 202, when determining the head posture information based on the facial key points in the image to be processed, the following steps may be included:

Step 2021: determine a head posture matrix based on the facial key points in the image to be processed, where the head posture matrix is used to represent the posture of the head in the image to be processed in the form of a matrix.

In one possible implementation, the head posture matrix can be calculated using the Golden Standard algorithm based on the pixel coordinates of the determined facial key points.

Step 2022: Determine the value of the target parameter based on the head posture matrix.

In a possible implementation, the value of the target parameter may be determined based on the element at the target position in the head posture matrix.

The head posture matrix may be a 3×3 matrix. Based on this, the value of the target parameter may be determined based on the head posture matrix by the following formula (1):

Among them, sy represents the target parameter, R _head represents the head posture matrix, R _head [0,0] represents the element in the 0th row and 0th column of the head posture matrix, and R _head [1,0] represents the element in the 1st row and 0th column of the head posture matrix.

Step 2023: Determine the head posture information based on the target parameter values and the head posture matrix.

Among them, the head posture information may include a pitch angle (Pitch), a rotational yaw angle (Yaw) and a rotational roll angle (Roll). In one embodiment, referring to FIG. 4 , FIG. 4 is a schematic diagram of a pitch angle, a rotational yaw angle and a rotational roll angle according to an embodiment of the present application. As shown in FIG. 4 , the angle of rotation of a person's head relative to the X-axis of the coordinate axis facing the camera is the pitch angle (Pitch), the angle of rotation of a person's head relative to the Y-axis of the coordinate axis facing the camera is the rotational yaw angle (Yaw), and the angle of rotation of a person's head relative to the Z-axis of the coordinate axis facing the camera is the rotational roll angle (Roll).

Determine the head posture information, that is, determine the pitch angle, the rotation yaw angle and the rotation roll angle. In some embodiments, when determining the head posture information based on the value of the target parameter and the head posture matrix, the following two implementation methods may be included:

In one possible implementation, when the value of the target parameter is greater than or equal to a set threshold, the pitch angle is determined based on the elements at the first set position and the second set position in the head posture matrix, the rotational yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the rotational roll angle is determined based on the elements at the fourth set position and the fifth set position in the head posture matrix.

In another possible implementation, when the value of the target parameter is less than a set threshold, the pitch angle is determined based on the elements at the sixth set position and the seventh set position in the head posture matrix, the rotational yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the value of the rotational roll angle is determined as the set angle value.

In one embodiment, the set threshold may be 1e-6 (i.e., 1*10 ^-6 ), the first set position may be the 2nd row and 1st column in the head posture matrix, the second set position may be the 2nd row and 2nd column in the head posture matrix, the third set position may be the 2nd row and 0th column in the head posture matrix, the fourth set position may be the 1st row and 0th column in the head posture matrix, the fifth set position may be the 0th row and 0th column in the head posture matrix, the sixth set position may be the 1st row and 2nd column in the head posture matrix, the seventh set position may be the 1st row and 1st column in the head posture matrix, and the set angle value may be 0 degrees.

For example, when the target parameter sy ≥ 1e-6, the pitch angle, the yaw angle, and the roll angle can be determined by the following formula (2):

Among them, _θx represents the pitch angle, _θy represents the yaw angle, _θz represents the roll angle, atan (also known as arctan) represents the inverse tangent value, _Rhead represents the head posture matrix, _Rhead [2,1] represents the element in the 2nd row and 1st column of the head posture matrix, _Rhead [2,2] represents the element in the 2nd row and 2nd column of the head posture matrix, _Rhead [2,0] represents the element in the 2nd row and 0th column of the head posture matrix, _Rhead [1,0] represents the element in the 1st row and 0th column of the head posture matrix, _Rhead [0,0] represents the element in the 0th row and 0th column of the head posture matrix, and sy represents the target parameter.

When the target parameter sy<1e-6, the pitch angle, yaw angle and roll angle can be determined by the following formula (3):

Among them, _θx represents the pitch angle, _θy represents the yaw angle, _θz represents the roll angle, atan (also known as arctan) represents the inverse tangent value, _Rhead represents the head posture matrix, _Rhead [1,2] represents the element in the first row and second column of the head posture matrix, _Rhead [1,1] represents the element in the first row and first column of the head posture matrix, _Rhead [2,0] represents the element in the second row and zero column of the head posture matrix, and sy represents the target parameter.

The head posture information can be determined through the above process. In addition, the head posture information and the head posture matrix can be converted to each other. The conversion relationship can be seen in the following formula (4):

Among them, R _head represents the head posture matrix, θ _x represents the pitch angle, θ _y represents the rotation yaw angle, θ _z represents the rotation roll angle, sin represents the sine value, and cos represents the cosine value.

After the head posture information is determined through the above process, step 203 can be used to determine the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed.

Among them, the pixel coordinates are two-dimensional coordinates including abscissas and ordinates, that is, the pixel coordinates of the left eye include the abscissas and ordinates of the left eye, and the pixel coordinates of the right eye include the abscissas and ordinates of the right eye. The three-dimensional coordinates of the human eye include the abscissas, ordinates, and depth coordinates of the left eye and the abscissas, ordinates, and depth coordinates of the right eye. Based on this, in some embodiments, for step 203, when determining the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the following steps may be included:

Step 2031: Determine the depth coordinates of the center points of the two eyes based on the pupil distance of the human eye, camera parameters, head posture information, pixel coordinates of the left eye and pixel coordinates of the right eye in the image to be processed.

The camera parameters can be in matrix form, that is, the camera parameters can be a camera parameter matrix. For example, the camera parameters can be recorded as

Optionally, the depth coordinates of the center points of the two eyes can be determined based on the pupil distance of the human eye, camera parameters, head posture information, pixel coordinates of the left eye and pixel coordinates of the right eye in the image to be processed by the following formula (5):

Among them, _zc represents the depth coordinate of the center point of the two eyes, d represents the pupil distance of the human eyes in the image to be processed, _ur represents the horizontal coordinate of the right eye, _ul represents the horizontal coordinate of the left eye, _vr represents the vertical coordinate of the right eye, _vl represents the vertical coordinate of the left eye, _fu , _fuv , _fvu , _fv , _u0 , and _v0 are all elements included in the camera parameter matrix, _Cy is also cos( _θy ), _Cz is also cos( _θz ), and _Sy is also sin( _θy ).

Step 2032: Based on the pupil distance of the human eyes, the depth coordinates of the center points of both eyes, and the head posture matrix in the image to be processed, the depth coordinates of the left eye and the depth coordinates of the right eye are determined respectively.

In one possible implementation, the depth coordinates of the left eye and the right eye can be determined based on the pupil distance of the human eye in the image to be processed, the depth coordinates of the center points of the two eyes, and the elements at the first set position, the second set position, and the third set position in the head posture matrix.

For example, the depth coordinates of the left eye and the right eye can be determined based on the pupil distance of the human eye in the image to be processed, the depth coordinates of the center points of the two eyes, and the elements at the first set position, the second set position, and the third set position in the head posture matrix by the following formula (6):

Among them, z _l represents the depth coordinate of the left eye, z _r represents the depth coordinate of the right eye, d represents the pupil distance of the human eye in the image to be processed, z _c represents the depth coordinate of the center point of the two eyes, R _head [2,1] represents the element at the first set position (row 2, column 1) in the head posture matrix, R _head [2,2] represents the element at the second set position (row 2, column 2) in the head posture matrix, and R _head [2,0] represents the element at the third set position (row 2, column 0) in the head posture matrix.

Step 2033: Determine the horizontal coordinate and vertical coordinate of the left eye, and the horizontal coordinate and vertical coordinate of the right eye based on the camera parameters, the pixel coordinates of the left eye, the pixel coordinates of the right eye, the depth coordinates of the left eye, and the depth coordinates of the right eye.

In one possible implementation, the target three-dimensional coordinates can be determined based on camera parameters, pixel coordinates of the left eye and pixel coordinates of the right eye, and then the three-dimensional coordinates of the human eye can be determined based on the target three-dimensional coordinates, depth coordinates of the left eye and depth coordinates of the right eye.

The target three-dimensional coordinates may include the target three-dimensional coordinates corresponding to the left eye and the target three-dimensional coordinates corresponding to the right eye. Coordinates, optionally, the target three-dimensional coordinates can be determined based on the camera parameters, the pixel coordinates of the left eye and the pixel coordinates of the right eye by the following formula (7):

in, Indicates the three-dimensional coordinates of the target corresponding to the left eye, Indicates the three-dimensional coordinates of the target corresponding to the right eye, represents the camera parameters, _ur represents the horizontal coordinate of the right eye, _vr represents the vertical coordinate of the right eye, _ul represents the horizontal coordinate of the left eye, and _vl represents the vertical coordinate of the left eye.

Furthermore, the three-dimensional coordinates of the human eye can be determined based on the target three-dimensional coordinates, the depth coordinates of the left eye, and the depth coordinates of the right eye by the following formula (8):

in, Represents the 3D coordinates of the left eye in the 3D coordinates of the human eye, represents the three-dimensional coordinates of the right eye in the three-dimensional coordinates of the human eye, Indicates the three-dimensional coordinates of the target corresponding to the left eye, represents the target 3D coordinates corresponding to the right eye, z _l represents the depth coordinates of the left eye, and z _r represents the depth coordinates of the right eye.

Through the above process, the pixel coordinates of the left and right eyes can be corrected based on the head posture information to obtain the three-dimensional coordinates of the left and right eyes. The three-dimensional coordinates are more accurate than the pixel coordinates, thereby improving the accuracy of the human eye positioning process.

The human eye positioning method provided in this application is used to process each frame of the video to determine the Taking the three-dimensional coordinates of the human eye in each frame of the image as an example, refer to Figure 5, which is a flow chart of a human eye positioning method according to an embodiment of the present application. As shown in Figure 5, a frame of image in the video stream can be obtained, and then it is determined whether the processing process is in a tracking state. If the processing process is not in a tracking state, that is, no, the image has not been processed before this frame of image or the face is not detected in the processed image, then face detection is performed based on the frame of image just obtained, so that when a face is detected, the detected face positioning frame is used as the face positioning frame of other images after this frame of image in the video stream to achieve face tracking, and when no face is detected, the next frame of image in the video stream is continued to be obtained for processing until a face is detected in the acquired image. In addition, when the processing process is in a tracking state, that is, it indicates that the face positioning frame has been determined before processing this frame of image, then the determined face positioning frame can be directly used as the face positioning frame of this frame of image. After the face positioning frame is determined, face key point detection can be performed based on the image portion corresponding to the face positioning frame, so as to determine the head posture information based on the face key points, and then determine the three-dimensional coordinates of the human eye based on the head posture information until the video processing is completed.

FIG. 5 is merely a procedural description, wherein the implementation of each step can refer to the above embodiment and will not be repeated here.

The above process mainly introduces the process of correcting the pixel coordinates of the left and right eyes based on the head posture information to obtain the three-dimensional coordinates of the left and right eyes, so as to effectively deal with the situation where the face in the image captured by the camera is not facing the camera. However, in real scenes, the face may not be in the center of the image. To deal with this situation, the present application also provides the following processing process/method.

Optionally, after determining the head posture matrix based on the facial key points in the image to be processed, the head posture matrix can also be transformed into a coordinate system to reduce the impact of the face not being in the center of the image on the process of determining the three-dimensional coordinates of the eye.

It should be noted that the determined head posture matrix (for example, the head posture matrix determined by the gold standard algorithm) is relative to the local coordinate system of the head. When the face is not in the center of the image, there will be a rotation transformation of the coordinates between the local coordinate system and the global coordinate system. Based on this, the head posture matrix can be converted from the local coordinate system to the global coordinate system to reduce the impact of the face not being in the center of the image on the three-dimensional coordinate determination process of the human eye.

In some embodiments, after the head posture matrix is determined based on the facial key points in the image to be processed, the coordinate system conversion can be achieved through the following steps.

Step 1: Determine the three-dimensional coordinates of the center points of the two eyes based on the camera parameters, the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed.

In a possible implementation, the pixel coordinates of the center point of the two eyes can be determined based on the pixel coordinates of the left eye and the right eye in the image to be processed, and then the three-dimensional coordinates of the center point of the two eyes can be determined based on the camera parameters and the pixel coordinates of the center point of the two eyes. The pixel coordinates also include a horizontal coordinate and a vertical coordinate.

Optionally, the pixel coordinates of the center points of the two eyes may be determined based on the pixel coordinates of the left eye and the pixel coordinates of the right eye using the following formula (9):

Among them, _uc represents the horizontal coordinate of the center point of the two eyes, _vc represents the vertical coordinate of the center point of the two eyes, _ul represents the horizontal coordinate of the left eye, _vl represents the vertical coordinate of the left eye, _ur represents the horizontal coordinate of the right eye, and _vr represents the vertical coordinate of the right eye.

After the pixel coordinates of the center points of the two eyes are determined by the above formula (9), the three-dimensional coordinates of the center points of the two eyes can be determined based on the camera parameters and the pixel coordinates of the center points of the two eyes by the following formulas (10) and (11).

in, Represents the three-dimensional coordinates of the center points of the two eyes, represents the camera parameters, _uc represents the horizontal coordinate of the center point of the two eyes, and _vc represents the vertical coordinate of the center point of the two eyes. That is, the homogeneous coordinates of the center points of the two eyes.

Step 2: Based on the three-dimensional coordinates of the center points of the two eyes, determine the rotation matrix, which is used to indicate the mapping relationship between the local coordinate system and the global coordinate system.

In a possible implementation, each column vector in the rotation matrix may be determined based on the three-dimensional coordinates of the center points of the two eyes, so as to determine the rotation matrix.

When the head posture matrix can be a 3×3 matrix, the rotation matrix is also a 3×3 matrix. Therefore, three column vectors can be determined based on the three-dimensional coordinates of the center points of the two eyes to achieve the determination of the rotation matrix.

Optionally, the rotation matrix can be determined by the following formulas (12) to (16):


C ₂ ＝C ₂ ′/||C ₂ ′|| ₂ (14)
C ₁ = cross(C ₂ , C ₃ ) (15)
R _local = [C ₁ C ₂ C ₃ ] (16)

Where R _local represents the rotation matrix, C ₁ , C ₂ , and C ₃ are column vectors in the rotation matrix respectively. Represents the three-dimensional coordinates of the center points of the two eyes, x, y, z are the abscissa, ordinate, and depth coordinates of the three-dimensional coordinates of the center points of the two eyes, respectively, and cross represents the vector cross product.

Step 3: Based on the rotation matrix and the head posture matrix, determine the target head posture matrix corresponding to the head posture matrix in the global coordinate system.

In a possible implementation, the head posture matrix may be rotationally transformed based on the rotation matrix to obtain a target head posture matrix corresponding to the head posture matrix in a global coordinate system.

Optionally, the target head posture matrix corresponding to the head posture matrix in the global coordinate system can be determined based on the rotation matrix and the head posture matrix by the following formula (17):
R _head ′＝R _local R _head (17)

Among them, R _head 'represents the target head posture matrix, R _local represents the rotation matrix, and R _head represents the rotation matrix.

After the head posture matrix is converted into a target head posture matrix in the global coordinate system, when the three-dimensional coordinates of the human eye are determined based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed through step 203, the target head posture information can be determined based on the target head posture matrix, thereby determining the three-dimensional coordinates of the human eye based on the pupil distance of the human eye, the camera parameters, the target head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye.

The process of determining the target head posture information based on the target head posture matrix can refer to the above formula (4) to convert the target head posture matrix into the target head posture information. The process of determining the three-dimensional coordinates of the human eye based on the human eye pupil distance, camera parameters, target head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye is the same as the implementation process of step 203, which will not be repeated here.

The above embodiments mainly introduce the implementation process of the human eye positioning method provided by the present application. In order to better understand the human eye positioning method provided by the present application, taking the application of the human eye positioning method provided by the present application to the ranging scenario as an example, after determining the three-dimensional coordinates of the human eye, the distance between the human eye and the camera can be determined based on the three-dimensional coordinates of the human eye, so that when the distance is less than the set distance threshold, a prompt message is sent, and the prompt message is used to prompt the user to increase the distance between the human eye and the camera to help the user prevent myopia.

When determining the distance between the human eye and the camera based on the three-dimensional coordinates of the human eye, the depth coordinate in the three-dimensional vertical coordinate of the human eye can be directly used as the distance between the human eye and the camera.

Optionally, the set distance threshold can be an arbitrary value. The present application does not limit the specific value of the set distance threshold. When the determined distance between the human eye and the camera is less than the set distance threshold, the computing device can prompt the user by sending a prompt message.

Among them, the prompt information can be voice prompt information, text prompt information, etc., and the present application does not limit the specific type of the prompt information.

The embodiment of the present application further proposes a human eye positioning device. Referring to FIG. 6 , FIG. 6 is a schematic diagram of the structure of a human eye positioning device according to the embodiment of the present application. As shown in FIG. 6 , the device includes:

The detection module 601 is used to detect key points of the face on the image to be processed including the face, and obtain the key points of the face in the image to be processed;

A first determination module 602 is used to determine head posture information based on key points of a human face in an image to be processed, where the head posture information is used to indicate a deflection angle between a human head and a preset direction;

The second determination module 603 is used to determine the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed.

In some embodiments of the present application, the first determination module 602, when used to determine the head posture information based on the facial key points in the image to be processed, is used to:

Based on the facial key points in the image to be processed, a head posture matrix is determined, where the head posture matrix is used to represent the posture of the head in the image to be processed in the form of a matrix;

Based on the head posture matrix, determine the value of the target parameter;

Based on the values of the target parameters and the head posture matrix, the head posture information is determined.

In some embodiments of the present application, the head posture information includes a pitch angle, a rotational yaw angle, and a rotational roll angle;

The first determination module 602, when used to determine the head posture information based on the value of the target parameter and the head posture matrix, is used to:

When the value of the target parameter is greater than or equal to the set threshold, the pitch angle is determined based on the elements at the first set position and the second set position in the head posture matrix, the rotational yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the rotational roll angle is determined based on the elements at the fourth set position and the fifth set position in the head posture matrix;

When the value of the target parameter is less than the set threshold, the pitch angle is determined based on the elements at the sixth set position and the seventh set position in the head posture matrix, the rotational yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the value of the rotational roll angle is determined as the set angle value.

In some embodiments of the present application, the pixel coordinates are two-dimensional coordinates including abscissas and ordinates, and the three-dimensional coordinates of the human eye include abscissas, ordinates, and depth coordinates of the left eye and abscissas, ordinates, and depth coordinates of the right eye;

The second determination module 603, when used to determine the three-dimensional coordinates of the human eye based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, is used to:

Determine the depth coordinates of the center points of the two eyes based on the pupil distance, camera parameters, head posture information, the pixel coordinates of the left eye and the pixel coordinates of the right eye;

Based on the pupil distance of the human eye, the head posture matrix, and the depth coordinates of the center points of both eyes, the depth coordinates of the left eye and the right eye are determined respectively;

Based on the camera parameters, the pixel coordinates of the left eye, the pixel coordinates of the right eye, the depth coordinates of the left eye, and the depth coordinates of the right eye, the abscissa and ordinate of the left eye, and the abscissa and ordinate of the right eye are determined.

In some embodiments of the present application, the device further comprises:

A third determination module is used to determine the three-dimensional coordinates of the center points of the two eyes based on the camera parameters, the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed;

The third determination module is further used to determine a rotation matrix based on the three-dimensional coordinates of the center points of the two eyes, where the rotation matrix is used to indicate a mapping relationship between the local coordinate system and the global coordinate system;

The third determination module is also used to determine the target head posture matrix corresponding to the head posture matrix in the global coordinate system based on the rotation matrix and the head posture matrix.

In some embodiments of the present application, the second determination module 603, when used to determine the three-dimensional coordinates of the human eye based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, is used to:

Based on the target head posture matrix, determine the target head posture information;

The three-dimensional coordinates of the human eye are determined based on the human eye pupil distance, camera parameters, target head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye.

In some embodiments of the present application, the detection module 601, when used to detect facial key points on an image to be processed including a face and obtain facial key points in the image to be processed, is used to:

Performing face detection based on the image to be processed to determine a face positioning frame in the image to be processed;

The facial key points are detected based on the image portion corresponding to the face positioning frame to obtain the facial key points in the image to be processed.

In some embodiments of the present application, the device further comprises:

A fourth determination module, used to determine the distance between the human eye and the camera based on the three-dimensional coordinates of the human eye;

The sending module is used to send a prompt message when the distance is less than a set distance threshold, and the prompt message is used to prompt the user to increase the distance between the human eye and the camera.

The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

The present application also provides a computing device, see Figure 7, which is a schematic diagram of the structure of a computing device according to an embodiment of the present application. As shown in Figure 7, the computing device includes a processor 710, a memory 720 and a network interface 730, the memory 720 is used to store computer program codes that can be run on the processor 710, the processor 710 is used to implement the human eye positioning method provided in any embodiment of the present application when executing the computer program code, and the network interface 730 is used to implement input and output functions. In more possible implementations, the computing device may also include other hardware, which is not limited in the present application.

The present application also provides a computer-readable storage medium, which can be in various forms. For example, in different examples, the computer-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof. In particular, the computer-readable medium can also be paper or other suitable medium capable of printing programs. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the human eye positioning method provided in any embodiment of the present application is implemented.

The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the human eye positioning method provided in any embodiment of the present application.

In the present application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the disclosure disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims (12)

A method for locating human eyes, comprising: Performing facial key point detection on the image to be processed including the face to obtain the facial key points in the image to be processed; Based on the key points of the human face in the image to be processed, determining head posture information, wherein the head posture information is used to indicate a deflection angle between the human head and a preset direction; Based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the right eye in the image to be processed, the three-dimensional coordinates of the human eye are determined, and the preset direction is the direction facing the camera. The method according to claim 1, wherein determining the head posture information based on the facial key points in the image to be processed comprises: Based on the facial key points in the image to be processed, determining a head posture matrix, wherein the head posture matrix is used to represent the posture of the head in the image to be processed in the form of a matrix; Based on the head posture matrix, determining the value of the target parameter; The head posture information is determined based on the value of the target parameter and the head posture matrix. The method according to claim 2, wherein the head posture information includes a pitch angle, a rotational yaw angle, and a rotational roll angle; The determining the head posture information based on the value of the target parameter and the head posture matrix includes: In the case where the value of the target parameter is greater than or equal to the set threshold, determining the pitch angle based on the elements at the first set position and the second set position in the head posture matrix, determining the rotational yaw angle based on the elements at the third set position in the head posture matrix and the value of the target parameter, and determining the rotational roll angle based on the elements at the fourth set position and the fifth set position in the head posture matrix; When the value of the target parameter is less than a set threshold, the pitch angle is determined based on the elements at the sixth set position and the seventh set position in the head posture matrix, the rotational yaw angle is determined based on the elements at the third set position in the head posture matrix and the value of the target parameter, and the value of the rotational roll angle is determined as the set angle value. The method according to claim 2 or 3, wherein the pixel coordinates are two-dimensional coordinates including abscissas and ordinates, and the three-dimensional coordinates of the human eye include abscissas, ordinates, and depth coordinates of the left eye and abscissas, ordinates, and depth coordinates of the right eye; The method of determining the three-dimensional coordinates of the human eye based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed includes: Determine the depth coordinates of the center points of the two eyes based on the human pupil distance, the camera parameters, the head posture information, the pixel coordinates of the left eye and the pixel coordinates of the right eye; Based on the human pupil distance, the head posture matrix, and the depth coordinates of the center points of the two eyes, respectively determine the depth coordinates of the left eye and the depth coordinates of the right eye; Based on the camera parameters, the pixel coordinates of the left eye, the pixel coordinates of the right eye, the depth coordinates of the left eye, and the depth coordinates of the right eye, the abscissa and ordinate of the left eye, and the abscissa and ordinate of the right eye are determined. The method according to any one of claims 2 to 4, wherein after determining the head posture matrix based on the facial key points in the image to be processed, the method further comprises: Determine the three-dimensional coordinates of the center points of the two eyes based on the camera parameters, the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed; Based on the three-dimensional coordinates of the center points of the two eyes, a rotation matrix is determined, where the rotation matrix is used to indicate a mapping relationship between a local coordinate system and a global coordinate system; Based on the rotation matrix and the head posture matrix, a target head posture matrix corresponding to the head posture matrix in the global coordinate system is determined. The method according to claim 5, wherein the determining the three-dimensional coordinates of the human eye based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed comprises: Based on the target head posture matrix, determining target head posture information; The three-dimensional coordinates of the human eye are determined based on the human eye pupil distance, the camera parameters, the target head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye. The method according to any one of claims 1 to 6, wherein the step of performing facial key point detection on the image to be processed including a face to obtain the facial key points in the image to be processed comprises: Performing face detection based on the image to be processed to determine a face positioning frame in the image to be processed; Facial key points are detected based on the image portion corresponding to the face positioning frame to obtain facial key points in the image to be processed. The method according to any one of claims 1 to 7, wherein after determining the three-dimensional coordinates of the human eye based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the method further comprises: Based on the three-dimensional coordinates of the human eye, determining the distance between the human eye and the camera; When the distance is less than a set distance threshold, a prompt message is sent, where the prompt message is used to prompt the user to increase the distance between the human eye and the camera. According to the method according to any one of claims 1 to 8, before performing facial key point detection on the image to be processed including the face, it also includes: detecting the quality of the face in the image to be processed to filter out faces with poor image quality. A human eye positioning device, comprising: A detection module, used to detect key points of human faces on the image to be processed, so as to obtain key points of human faces in the image to be processed; A first determination module, configured to determine head posture information based on key points of a human face in the image to be processed, wherein the head posture information is used to indicate a deflection angle between a human head and a preset direction; The second determination module is used to determine the three-dimensional coordinates of the human eye based on the pupil distance of the human eye in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed. A computing device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the operations performed by the human eye positioning method as described in any one of claims 1 to 9 are implemented. A non-volatile computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the operation performed by the human eye positioning method according to any one of claims 1 to 9 is implemented.