WO2023010301A1 - Line-of-sight detection method and apparatus, and eyeball model modeling method and apparatus - Google Patents

Abstract

The present application relates to the field of artificial intelligence. Provided are a line-of-sight detection method and apparatus, and an eyeball model modeling method and apparatus. The detection method comprises: acquiring an image to be processed, wherein the image to be processed comprises an eye region portion of a target object; and minimizing the difference between a line-of-sight estimation image and the image to be processed, so as to obtain the line of sight of the target object, wherein the line-of-sight estimation image is obtained by means of using an eye region model to cover the eye region portion, the eye region model is obtained by using a three-dimensional eyeball model, and the eyeball model is obtained by means of using a texture image of an eyeball to render an eyeball grid. The modeling means for an eyeball model in the solution is simple and the eyeball model is easily acquired; therefore, the complexity is significantly reduced, the acquisition cost is reduced, and the eyeball model is easily imitated and used. In addition, the eye region model is a universal model, and can therefore be applied to all line-of-sight detection scenarios and has a good transportability.

Description

Line-of-sight detection method, eyeball model modeling method and device thereof technical field

The embodiments of the present application relate to the field of artificial intelligence, and more specifically, relate to a line of sight detection method and device thereof.

Background technique

Human sight is the most important interactive information besides language. This information is valuable to a wide variety of fields, including psychology, sociology, marketing, robotics, human-computer interface, and more. In the human-computer interaction in the vehicle field, eye-tracking technology helps to understand the occupant's behavior and intention, and has unique application value.

In some solutions, it is often necessary to estimate the starting point and direction of the line of sight in three-dimensional space based on the depth camera, so as to determine the realization direction of the person. But this eye-tracking method based on depth camera is too complicated and has poor portability. Specifically, each line-of-sight detection system includes equipment such as a conventional camera, a depth camera, and a display screen, as well as a corresponding detection model, and the line-of-sight detection system requires training of the detection model and parameter calibration of the device before use. Specifically include: it is necessary to use the camera in the system to collect images for training the detection model, and the training of the detection model will require a large number of labeled, high-definition eye images; it is also necessary to use the conventional camera, depth camera and display screen for parameter calibration. Therefore, both the detection model and the calibration parameters can only be applied to this line of sight detection system, and if it is switched to other line of sight detection systems, the above work needs to be repeated. Even just replacing a certain device in the system or changing the location of the device requires re-calibration of parameters and re-training of the detection model. That is to say, the complexity and cost of the above-mentioned schemes are too high, and the portability is too poor.

In other solutions, the line of sight direction can be extracted from images including people, but this method requires the assistance of an eye model, which is difficult to obtain, complex, and expensive to obtain, and is mostly kept secret by developers. Difficult to obtain, and even if obtained, difficult to imitate or use.

Therefore, how to reduce the complexity of line-of-sight detection is an urgent technical problem to be solved.

Contents of the invention

Embodiments of the present application provide a line of sight detection method, an eyeball model modeling method and a device thereof, which can reduce the complexity of line of sight detection.

In the first aspect, a line of sight detection method is provided, the method comprising: acquiring an image to be processed, the image to be processed includes the eye region of the target object; minimizing the difference between the estimated line of sight image and the image to be processed, to obtain the line of sight of the target object, the line of sight The estimated image is obtained by covering the eye region with the eye region model, the eye region model is obtained by using the three-dimensional eyeball model, and the eyeball model is obtained by rendering the eyeball mesh using the texture image of the eyeball.

In the technical solution of the present application, the line of sight direction is mainly determined by minimizing the difference between the estimated line of sight image and the image to be processed. The eye area model is obtained by using a three-dimensional eyeball model. The modeling method of the eyeball model is simple, It is easy to obtain, so the complexity is greatly reduced, the cost of acquisition is reduced, and it is easy to imitate and use. In addition, since the eye area model used to obtain the eye area estimation image is a general model, it can be applied to all line of sight detection scenarios. As long as the image to be processed including the eye area part can be provided, the method of the embodiment of the present application can be applied, and it can be transplanted Good performance and high universality. In addition, compared with the prior art, since it is no longer necessary to repeat the process of training the model and calibrating the equipment, the operation is simple, the time is shortened, and the cost is reduced.

It should be noted that the difference between the estimated line of sight image and the image to be processed can be understood as the difference between the two images, which may include key point reprojection differences, pixel differences, and the like. It is understandable that since the eye area is partially covered by the eye area model, and there are often some key point position differences and pixel differences between the eye area model and the eye area part, it is possible to reduce the difference between the two, Make the eye area model close to the eye area part.

With reference to the first aspect, in some implementation manners of the first aspect, the eyeball grid is obtained by using a three-dimensional human head model. Those skilled in the art can easily obtain various human head models, so eyeball grids are also easy to obtain.

In some implementation manners, the mapping relationship between the vertices of the eyeball mesh and the corresponding points in the texture image may be established, and the mapping relationship between the triangles of the eyeball mesh and the corresponding point intervals in the texture image may also be established. With reference to the first aspect, in some implementation manners of the first aspect, the eyeball model is obtained by rendering the eyeball mesh according to the mapping relationship between the eyeball mesh and the texture image.

In combination with the first aspect, in some implementations of the first aspect, the above-mentioned mapping relationship between the eyeball grid and the texture image can be a non-linear mapping relationship, which can make the eyeball model more natural, that is, closer to the characteristics of the real eyeball .

With reference to the first aspect, in some implementation manners of the first aspect, the difference between the estimated line of sight image and the image to be processed is represented by an energy function. When minimizing the difference between the estimated line of sight image and the image to be processed to obtain the line of sight of the target object, the following operations can be performed: minimizing the energy function to obtain attitude parameters, and the attitude parameters include the line of sight. The posture parameter can be understood as the posture of the target object, such as eyeball orientation, mouth opening and closing, or head posture, etc. Therefore, it can also be seen that the posture parameter includes the above-mentioned line of sight, that is, eyeball orientation.

With reference to the first aspect, in some implementation manners of the first aspect, when minimizing the energy function, one or more of the following parameters may also be obtained: face shape parameters, texture parameters or illumination parameters. That is, in the process of minimizing the energy function, other useful parameters can be provided besides the task of line-of-sight detection. The face parameter can be understood as the shape of the face of the target object, and the texture parameter can be understood as the texture of the target object, such as skin color, eyebrows, spots, etc., and the lighting parameters can include one or more of the following parameters: light direction, The type or color of the light source.

In a second aspect, a method for modeling an eyeball model is provided, and the modeling method includes: acquiring an eyeball grid and a texture image of the eyeball; rendering the eyeball grid by using the texture image to obtain an eyeball model. The modeling method is simple and easy to operate, and the eyeball model is simple in modeling and easy to obtain, so the complexity is greatly reduced, the acquisition cost is reduced, and it is easy to imitate and use. In addition, this modeling method can obtain a general eyeball model, which is established using vertices, topological structures and textures, so when used for line of sight detection, these parameters can bring differences such as pixels on the image, while By minimizing the difference of the image, the parameters of the above-mentioned eyeball models are inversely deduced. Therefore, using the eyeball model obtained by the above method to perform line-of-sight detection can have good portability.

With reference to the second aspect, in some implementation manners of the second aspect, the eyeball grid is obtained by using a three-dimensional human head model. Those skilled in the art can easily obtain various human head models, so eyeball grids are also easy to obtain.

In combination with the second aspect, in some implementations of the second aspect, when using the texture image to render the eyeball grid, a mapping relationship between the eyeball grid and the texture image can be established; according to the mapping relationship, the texture image The texture is rendered onto the eye mesh.

With reference to the second aspect, in some implementation manners of the second aspect, the foregoing mapping relationship is nonlinear. This can make the eyeball model more natural, that is, closer to the characteristics of real eyeballs.

With reference to the second aspect, in some implementation manners of the second aspect, the eyeball model may also be placed in the head model, where the texture of the head model is in a different quadrant from the texture of the eyeball model. This results in a head model with clear eye textures.

In a third aspect, a line-of-sight detection device is provided, and the device includes a unit for performing the method in any one implementation manner of the above-mentioned first aspect.

In a fourth aspect, an eyeball model modeling device is provided, and the device includes a unit for performing the method in any one of the above-mentioned implementation manners of the second aspect.

According to a fifth aspect, there is provided a computing device, which includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is used for Execute the method in any one of the implementation manners in the first aspect or the second aspect. The device may be a vehicle-mounted terminal, a host computer, a computer, a server, a cloud device, or any other device or system that needs line-of-sight detection, or it may be a device installed in the above-mentioned device or system. The device can also be a chip.

In a sixth aspect, a computer-readable medium is provided, the computer-readable medium stores program code for execution by a device, and the program code includes a method for executing the method in any one of the implementation manners of the first aspect or the second aspect .

In a seventh aspect, a computer program product including instructions is provided, and when the computer program product is run on a computer, it causes the computer to execute the method in any one of the above-mentioned first aspect or the second aspect.

In an eighth aspect, a chip is provided, the chip includes a processor and a data interface, and the processor reads instructions stored on the memory through the data interface, and executes any one of the above-mentioned first aspect or second aspect method in the implementation.

Optionally, as an implementation manner, the chip may further include a memory, the memory stores instructions, the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the The processor is configured to execute the method in any one of the implementation manners of the first aspect or the second aspect.

In this application, the modeling method of the eyeball model used is simple, and the eyeball model is easy to obtain and use, so the complexity of the line of sight estimation method is greatly reduced, and the cost of obtaining the eyeball model is reduced. In addition, since the eye area model used to obtain the eye area estimation image is a general model, it can be applied to all line of sight detection scenarios. As long as the image to be processed including the eye area can be provided, the method of the embodiment of the present application can be applied. Therefore, The solution of the present application also has the advantages of good portability and high universality. This application also proposes that the eyeball mesh can be obtained by using the 3D human head model with insufficient details but which is easy to obtain, and then render the eyeball mesh to obtain the eyeball model. This is a simple method of obtaining the eyeball model, which can Further reduce costs and simplify the modeling process. Establishing the mapping relationship between the eyeball grid and the texture image can make the rendering effect better. The nonlinear mapping relationship can further improve the rendering effect, making the eyeball model more natural, that is, closer to the characteristics of real eyeballs.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario of an embodiment of the present application.

Fig. 2 is a schematic flow chart of a sight line detection method according to an embodiment of the present application.

Fig. 3 is a schematic diagram of a human eyeball structure and an eyeball model according to an embodiment of the present application.

FIG. 4 is a schematic diagram of a line of sight detection process according to an embodiment of the present application.

Fig. 5 is a schematic flow chart of an eyeball model modeling method according to an embodiment of the present application.

Fig. 6 is a schematic diagram of a human head model and an eyeball grid according to an embodiment of the present application.

Fig. 7 is a schematic diagram of a texture image according to an embodiment of the present application.

Fig. 8 is a schematic diagram of the eyeball model of the embodiment of the present application.

Fig. 9 is a schematic diagram of a human head model including an eyeball model according to an embodiment of the present application.

Fig. 10 is a schematic diagram of an applicable scenario of the embodiment of the present application.

Fig. 11 is a schematic block diagram of a line of sight detection device according to an embodiment of the present application.

FIG. 12 is a schematic diagram of a hardware structure of a line of sight detection device according to an embodiment of the present application.

Fig. 13 is a schematic block diagram of an eyeball model modeling device according to an embodiment of the present application.

Fig. 14 is a schematic diagram of the hardware structure of the eyeball model modeling device according to the embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

The line-of-sight detection solution of the embodiments of the present application can be used in any occasion where line-of-sight direction needs to be determined, for example, it can be used in the field of smart cars, monitoring fields, or tracking and shooting. FIG. 1 is a schematic diagram of an application scenario of an embodiment of the present application. As shown in Figure 1, the line of sight of the target object can be obtained by inputting the image to be processed into the line of sight detection device. The image to be processed includes the eye region (left eyeball area or right eyeball area) of the target object, and the target object can be a person or a person. It may be an animal, and the line of sight of the target object is the line of sight direction of the target object. That is to say, the line-of-sight detection device is used to process the image to be processed to obtain the line-of-sight direction of the target object in the image to be processed. Assuming that the image to be processed comes from the camera on the vehicle and the target object is the driver, the driver's intention can be inferred according to the driver's gaze direction, thereby assisting the driver in controlling the vehicle. For another example, assuming that the image to be processed comes from a surveillance camera, and the target object is a key person such as a person with abnormal behavior captured, it is possible to lock what the key person is looking at according to the direction of the key person's line of sight, so as to facilitate inferring the key person next move. For another example, suppose that the image to be processed comes from a smart camera that can track and shoot, and the target object can be a person or an animal. At this time, the shooting angle can be adjusted by continuously obtaining the line of sight of the target object, so as to improve Accurately capture the activity of the target object, such as tracking the natural movement of animals.

Fig. 2 is a schematic flow chart of a sight line detection method according to an embodiment of the present application. Each step shown in FIG. 2 will be introduced below.

201. Acquire an image to be processed, where the image to be processed includes an eye region of a target object.

The eye area part is the part including the eyes, and the eye area part may include the left eyeball, may also include the right eyeball, or may include two eyeballs.

It should be understood that in this embodiment of the present application, the target object may be a human or an animal, and there is no limitation. To facilitate the understanding of the solution, the following mainly introduces that the target object is a human as an example.

Optionally, the image to be processed may be captured by sensing equipment such as a camera or a video camera; the image to be processed may also be read from a storage device; or obtained from a network such as the Internet or the Internet of Vehicles using a communication interface The image to be processed above.

202. Minimize the difference between the estimated line of sight image and the image to be processed to obtain the line of sight of the target object, where the estimated line of sight image is obtained by covering the eye area part with an eye area model.

It should be understood that since the image to be processed is two-dimensional, the eye region model is also two-dimensional.

After the eye area part of the image to be processed is covered with the eye area model, there must be some differences between the estimated line of sight image and the image to be processed. Therefore, the line of sight estimated image can be adjusted by minimizing the difference between the two, so that The difference between the line of sight estimated image and the image to be processed is as small as possible.

Optionally, the above-mentioned eye region model may be obtained by using a three-dimensional eyeball model, and the eyeball model may be obtained by rendering an eyeball mesh using a texture image of the eyeball. For the introduction of the modeling method of the eyeball model, reference may be made to the relevant content in FIG. 5 , and details will not be elaborated here.

The eyeball model can be understood as a three-dimensional model for representing the eyeball and including information such as the structure, texture, color and shape of the eyeball. In order to facilitate the understanding of the eyeball model, a human being is taken as an example and introduced in conjunction with FIG. 3 . FIG. 3 is a schematic diagram of a human eyeball structure and an eyeball model according to an embodiment of the present application. As shown in Figure 3, (a) in Figure 3 is the structure of the human eyeball. It can be seen that the eyeball is composed of two spherical structures of different sizes, which can be represented by structure #1 and structure #2 respectively for easy understanding, where , the larger spherical structure part (i.e. structure #1) is mainly the vitreous body, and also includes the retina, fovea, optic nerve, etc.; the smaller spherical structure part (i.e. structure #2) is mainly the anterior chamber, and the lens, ciliary body, iris, cornea, pupil, etc. (a) in FIG. 3 also shows the representation method of the line of sight direction, that is, the direction of the arrow from the fovea through the optical center, the center of the pupil to the outside of the eyeball. (b) in Figure 3 is the human eyeball model. It can be seen that the eyeball model is also two spherical structures of different sizes, of which the larger spherical structure corresponds to the above-mentioned structure #1, and the smaller spherical structure corresponds to It is the above structure #2. It can also be seen from (b) in Figure 3 that both structure #1 and structure #2 include relatively rich information such as shape, color, texture, etc. These eyeball information are also similar to those in Figure 3 ( The above information of each eyeball component in a) corresponds to.

Optionally, the eyeball grid can be obtained by using a three-dimensional human head model. Those skilled in the art can easily obtain various human head models, so eyeball grids are also easy to obtain.

Optionally, the above-mentioned mapping relationship between the eyeball grid and the texture image may be a non-linear mapping relationship, which can make the eyeball model more natural, that is, closer to the characteristics of real eyeballs. Since the closer to the center of the eyeball (pupil), the information included is richer or can be understood as more details, such as denser textures, more color changes, etc., and it is also particularly important for the determination of the line of sight, and the farther away from the center of the eyeball The part of the pupil (pupil), such as the part of the vitreous body, contains relatively little information, such as relatively sparse texture and less color change, etc., and has little influence on the determination of the line of sight, so it is possible to establish a non-linear mapping relationship, focusing on The rendering of the pupil part weakens the rendering of the vitreous part, thereby effectively improving the accuracy of the eye model and improving the calculation efficiency. When such an eyeball model is used, the accuracy of line of sight detection can be effectively improved.

The following describes the difference between the minimized line of sight estimation image and the image to be processed. This difference can be understood as the difference between the two images, which can include key point reprojection differences, pixel differences, etc. It can be understood that due to the eye area part It is covered by the eye area model, and the eye area model and the eye area part often have some key point position differences, pixel differences, etc., so the difference between the two can be reduced to make the eye area model close to the eye area part.

In some implementation manners, the above-mentioned difference may be represented by an energy function, and the energy function is used to measure the above-mentioned difference. Assume that the parametric eye region model is: M=M(β,τ,θ), that is, M is a function of β, τ, and θ, where M represents the vertex position of the eyeball grid, and β represents the face shape parameter. τ represents a texture parameter, and θ represents an attitude parameter, so the above formula is equivalent to that the vertex position of the eye area model is a function of the face shape parameter, the texture parameter, and the attitude parameter. Assume that neural rendering is: I=NR(M,l,k)), that is, I is a function of M, l and k, where I is the rendered image, that is, the above-mentioned line-of-sight estimation image, and l represents Illumination parameters, k represents the camera parameters, then the above formula is equivalent to that the line of sight estimation image is a function of the eye area model, the illumination parameters and the camera parameters. Therefore, if the energy function is constructed, the energy function can be:

E＝ε _image (I _syn ,I _obs )

＝ε _image (NR(M,l,k),I _obs )

＝ε _image (NR(M(β,τ,θ),l,k),I _obs ),

Among them, I _obs represents the difference of the position of the above key points, I _syn represents the difference of the above pixels, and ε _image represents the total difference between the line-of-sight estimation image and the image to be processed. That is to say, E is ultimately a function of β, τ, θ, l and k. Since the above three functions are all differentiable functions, E can be differentiated for β, τ, θ and l. Therefore, to minimize E One or more of the above parameters can be obtained: β, τ, θ or l, since k is a known parameter, it is not necessary to obtain k.

The various parameters mentioned above are introduced below. The posture parameter can be understood as the posture of the target object, such as eyeball orientation, mouth opening and closing, or head posture, etc. Therefore, it can also be seen that the posture parameter includes the above-mentioned line of sight, that is, eyeball orientation. The face shape parameter can be understood as the shape of the face of the target object, and the texture parameter can be understood as the texture of the target object, such as skin color, eyebrows, spots, etc., and the lighting parameters can include one or more of the following: light direction, light source type or Light source color.

It should be understood that to minimize the above energy function, all the above parameters may be obtained, but only some of the parameters may be obtained. For example, the attitude parameters can be obtained by minimizing the energy function. Since the attitude parameters include realization, obtaining the attitude parameters is equivalent to obtaining the line of sight of the target object. For another example, when the energy function is minimized, one or more of the following parameters are also obtained: face shape parameters, texture parameters or illumination parameters. That is, in the process of minimizing the energy function, other useful parameters can be provided besides the task of line-of-sight detection.

The method shown in Figure 2 mainly determines the line of sight direction by minimizing the difference between the estimated line of sight image and the image to be processed. The eye area model is obtained by using a three-dimensional eyeball model. The eyeball model only needs to use the most common modeling It can be obtained in a simple way, so the complexity is greatly reduced, the cost of acquisition is reduced, and it is easy to imitate and use. In addition, since the eye area model used to obtain the eye area estimation image is a general model, it can be applied to all line of sight detection scenarios. As long as the image to be processed including the eye area part can be provided, the method of the embodiment of the present application can be applied, and it can be transplanted Good performance and high universality. In addition, compared with the prior art, since it is no longer necessary to repeat the process of training the model and calibrating the equipment, the operation is simple, the time is shortened, and the cost is reduced.

FIG. 4 is a schematic diagram of a line of sight detection process according to an embodiment of the present application. FIG. 4 can be regarded as an example of the process of executing the method shown in FIG. 2 . As shown in FIG. 4 , A is an image to be processed, and the area a of the image to be processed including the eyes is enlarged, as shown in B. C is the image after covering the eye area with the eye area model, that is, a part of the line of sight estimation image. It can be seen that, except for the eye area model, other parts of C are consistent with B and A and remain unchanged. D is an image including the line of sight direction (ie, the arrow in the figure), that is, the adjusted line of sight estimated image obtained after minimizing the difference between the line of sight estimated image and the image to be processed, and D is obviously closer to B than C. That is to say, the method shown in Figure 2 is the process of obtaining D by minimizing the difference between C and B.

Fig. 5 is a schematic flow chart of an eyeball model modeling method according to an embodiment of the present application. Each step shown in FIG. 5 will be introduced below.

501. Acquire an eyeball grid and a texture image of the eyeball.

The eyeball grid can be understood as using a three-dimensional grid structure to represent the outline structure of the eyeball, and the texture image of the eyeball is used as an image including texture information of the eyeball.

Optionally, the eyeball grid can be extracted from a three-dimensional human head model. The human head model is a general three-dimensional human face model, which represents the human face with a fixed fixed-point number and a preset topology (triangular surface). However, the current face models are generally not clear enough in terms of eyeball texture, so they cannot be used for line of sight detection. Fig. 6 is the schematic diagram of human head model and eyeball grid of the embodiment of the present application. As shown in Figure 6, (a) in Figure 6 is the front view of the human head model, and (b) in Figure 6 is the front view of the eyeball grid extracted from the human head model shown in Figure 6 (a) Figure, (c) among Fig. 6 is the side view of above-mentioned eyeball grid. As an example, the eye mesh consists of 546 vertices and 1088 triangles.

Those skilled in the art can use existing eyeball texture images or draw eyeball texture images by themselves. Fig. 7 is a schematic diagram of a texture image according to an embodiment of the present application. As shown in Figure 7, these texture maps (a)-(c) all contain rich eyeball texture information, and the textures in several texture maps are different, such as pupil size, texture, color, etc.

502. Use the texture image to render the eyeball grid to obtain an eyeball model.

That is to say, the eyeball texture in the texture image can be rendered onto the eyeball grid to obtain a structure with the eyeball texture, which is the eyeball model, as shown in FIG. 8 , for example. Fig. 8 is a schematic diagram of the eyeball model of the embodiment of the present application. (a) in FIG. 8 is a front view of the eyeball model, and (b) in FIG. 8 is a side view of the eyeball model.

It should be noted that, in this embodiment of the application, rendering refers to using a renderer carried by various 3D production software such as Maya, 3ds Max, or Blender, or using an independent renderer such as RenderMan, Octane, V-Ray, or Arnold. The produced three-dimensional model (such as the above-mentioned eyeball mesh) is added with elements such as texture (such as the above-mentioned eyeball texture), binding, animation or lighting to obtain a more vivid model (such as the above-mentioned eyeball model) or the final display effect of the animation , is also the last important procedure in 3D production. It can be understood that rendering is a process of making a geometric model have various textures such as materials, colors, and lines, various lighting scenes, and actions.

Optionally, the following steps 502-1 and 502-2 may be used to execute step 502.

502-1. Establish a mapping relationship between the eyeball grid and the texture image.

502-2. Render the texture of the texture image onto the eye grid according to the above mapping relationship.

In some implementation manners, the mapping relationship between the vertices of the eyeball mesh and the corresponding points in the texture image may be established, and the mapping relationship between the triangles of the eyeball mesh and the corresponding point intervals in the texture image may also be established.

Taking an eyeball grid including 546 vertices and 1088 triangular faces as an example, the eyeball grid includes 2 antidiametrical points, 32 longitudes and 17 latitudes, and the intersection of longitudes and latitudes and the antidiametrical points are the above 546( 32*17+2) vertices. Therefore, the mapping relationship can be: two antidiametric points correspond to the center of the texture image, and the texture image sets multiple circles with different radii with the center as the dot, and multiple straight lines passing through the center of the circle, then these circles and these straight lines correspond to each other The above-mentioned latitude and longitude, the intersection of these circles and straight lines correspond to the vertices other than the anti-radial point at the above-mentioned place.

Optionally, in order to render the eyeball model more similar to the real eyeball, the radii of the above-mentioned circles may not be equally divided, and the included angles between the lines may not be completely equal, that is, the above-mentioned mapping relationship is non-linear.

The modeling method shown in Figure 5 is simple and easy to operate. The modeling method of the eyeball model is simple and easy to obtain, so the complexity is greatly reduced, the acquisition cost is reduced, and it is easy to imitate and use. In addition, the method shown in Figure 5 can obtain a general eyeball model, which is established using vertices, topological structures, and textures. Therefore, when used for line of sight detection, these parameters can bring differences such as pixels on the image, And by minimizing the difference of the image, the parameters of the above-mentioned eyeball models are inversely deduced. Therefore, using the eyeball model obtained by the method shown in FIG. 5 to perform line-of-sight detection can have good portability.

In some implementation manners, the method shown in FIG. 5 may further include step 503 .

503. Put the eyeball model into the head model to obtain the head model including the eyeball model.

Optionally, the texture of the eyeball model can be placed in a different quadrant from the texture of the human head model in step 501, the correspondence between the eyeball vertices of the human head model and the vertices of the eyeball model, and the texture of the eyeball vertices of the human head model can be established Mapping, you can put the eyeball model into the head model. For example, the texture of the human head model in step 501 can be placed in the third quadrant, and the texture of the eyeball model can be placed in the second quadrant.

However, it should be understood that in some cases, the head model may be the head model described in step 501 , but in other cases, the head model may also be other head models different from step 501 . Fig. 9 is a schematic diagram of a human head model including an eyeball model according to an embodiment of the present application. It can be seen from Figure 9 that the eyes of the human head model have clear details, so the human head model can be used in scenes that require clear eyeball textures such as line of sight detection. (a) in FIG. 9 is a front view of a human head model, and (b) in FIG. 9 is a front view of a human head model in various viewing directions. This is what the existing human head models do not have.

Fig. 10 is a schematic diagram of an applicable scenario of the embodiment of the present application. Fig. 10 is an example of applying the embodiment of the present application to a vehicle, and the vehicle may be an ordinary car, an electric car, a new energy vehicle, a truck, a passenger car, and other types of vehicles. As shown in FIG. 10 , an image including the driver can be captured by the camera installed at the front A of the vehicle, which is the image to be processed, and the target object in the image to be processed is the driver. The following are examples of two specific usage scenarios.

Scenario 1: The display wakes up from sleep. In this scenario, when the driver's line of sight is not on the display screen (such as display screen B in Figure 10), the display screen can be temporarily hidden or dormant, thereby reducing power consumption. When it is detected that the driver's line of sight is pointing to When the display screen is turned on, the display screen is woken up for display. That is, it is assumed that the driver's line of sight direction obtained by using the scheme of the embodiment of the present application is a display screen, as shown in (a) in Figure 10, the line of sight direction (the direction of the arrow PQ) is from the driver to the traffic light C, and now it can be seen The display screen B does not display (does not present an image); as shown in (b) in Figure 10, the line of sight (direction of the arrow MN) is directed from the driver to the display screen B, and at this time it can be inferred that the driver wants to see the display If the content on the screen is displayed, the display function of the display screen will be awakened.

Scenario 2: The enlarged display function of traffic lights. In the process of driving, it is often encountered that the traffic lights are too far away, so that the color of the traffic lights or the number of the timing cannot be seen clearly. Partial zoom-in is performed and displayed on the display screen B in real time, which is convenient for the driver to observe. For example, as shown in (a) in Figure 10, assuming that the line of sight (the direction of the arrow PQ) is from the driver pointing to the traffic light C out of the front window, at this time, it can be inferred that the driver wants to see the information of the traffic light clearly, then the The traffic lights are partially enlarged and displayed on the display screen B in real time, as shown in (b) in FIG. 10 .

However, it should be understood that Fig. 10 only gives two simple examples. In practice, the line-of-sight detection solution provided by the embodiment of the present application can be used in any scene where the driver's line-of-sight direction needs to be known, and can also be applied to any other objects that need to be known. The scenes in the direction of the subject's line of sight will not be listed one by one.

Fig. 11 is a schematic block diagram of a line of sight detection device according to an embodiment of the present application. The apparatus 2000 shown in FIG. 11 includes an acquisition unit 2001 and a processing unit 2002 .

The acquisition unit 2001 and the processing unit 2002 may be configured to execute the line of sight detection method of the embodiment of the present application. Specifically, the acquisition unit 2001 may execute the above step 201, and the processing unit 2002 may execute the above step 202.

It should be understood that the processing unit 2002 in the above device 2000 may be equivalent to the processor 3002 in the device 3000 hereinafter.

FIG. 12 is a schematic diagram of a hardware structure of a line of sight detection device according to an embodiment of the present application. The line of sight detection apparatus 3000 shown in FIG. 12 (the apparatus 3000 may specifically be a computer device) includes a memory 3001 , a processor 3002 , a communication interface 3003 and a bus 3004 . Wherein, the memory 3001 , the processor 3002 , and the communication interface 3003 are connected to each other through a bus 3004 .

The memory 3001 may be a read only memory (read only memory, ROM), a static storage device, a dynamic storage device or a random access memory (random access memory, RAM). The memory 3001 may store a program, and when the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are used to execute each step of the line of sight detection method according to the embodiment of the present application.

The processor 3002 may be a general-purpose central processing unit (central processing unit, CPU), a microprocessor, an application specific integrated circuit (application specific integrated circuit, ASIC), a graphics processing unit (graphics processing unit, GPU) or one or more The integrated circuit is used to execute related programs, so as to realize the functions required by the units in the line of sight detection device of the embodiment of the present application, or execute the line of sight detection method of the method embodiment of the present application.

The processor 3002 may also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the line of sight detection method of the present application may be completed by an integrated logic circuit of hardware in the processor 3002 or instructions in the form of software. The above-mentioned processor 3002 can also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistors Logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 3001, and the processor 3002 reads the information in the memory 3001, and combines its hardware to complete the functions required by the units included in the sight detection device of the embodiment of the present application, or execute the sight detection of the method embodiment of the present application method.

The communication interface 3003 implements communication between the apparatus 3000 and other devices or communication networks by using a transceiver device such as but not limited to a transceiver. For example, the image to be processed may be obtained through the communication interface 3003 .

The bus 3004 may include a pathway for transferring information between various components of the device 3000 (eg, memory 3001 , processor 3002 , communication interface 3003 ).

Fig. 13 is a schematic block diagram of an eyeball model modeling device according to an embodiment of the present application. The apparatus 4000 shown in FIG. 13 includes an acquisition unit 4001 and a processing unit 4002 .

The acquisition unit 4001 and the processing unit 4002 can be used to implement the eyeball model modeling method of the embodiment of the present application. Specifically, the acquisition unit 4001 can perform the above step 501, and the processing unit 4002 can perform the above step 502. The processing unit 4002 may also execute the above step 503 .

The acquiring unit 4001 may also be integrated in the processing unit 4002.

It should be understood that the processing unit 4002 in the above-mentioned device 4000 may be equivalent to the processor 5002 in the device 5000 hereinafter.

Fig. 14 is a schematic diagram of the hardware structure of the eyeball model modeling device according to the embodiment of the present application. The apparatus 5000 shown in FIG. 16 (the apparatus 5000 may specifically be a computer device) includes a memory 5001 , a processor 5002 , a communication interface 5003 and a bus 5004 . Wherein, the memory 5001 , the processor 5002 , and the communication interface 5003 are connected to each other through a bus 5004 .

The memory 5001 may be a ROM, a static storage device, a dynamic storage device or a RAM. The memory 5001 can store programs, and when the programs stored in the memory 5001 are executed by the processor 5002, the processor 5002 and the communication interface 5003 are used to execute each step of the eyeball model modeling method of the embodiment of the present application.

The processor 5002 may adopt a CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits for executing related programs, so as to realize the functions required by the units in the eyeball model modeling device of the embodiment of the present application, Or execute the eyeball model modeling method of the method embodiment of the present application.

The processor 5002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the gaze detection network of the present application may be completed by an integrated logic circuit of hardware in the processor 5002 or instructions in the form of software. The aforementioned processor 5002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 5001, and the processor 5002 reads the information in the memory 5001, and combines its hardware to complete the functions required by the units included in the eyeball model modeling device of the embodiment of the application, or execute the method embodiment of the application Methods.

The communication interface 5003 implements communication between the apparatus 5000 and other devices or communication networks by using a transceiver device such as but not limited to a transceiver. For example, the above-mentioned eyeball grid and texture images can be obtained through the communication interface 5003 .

The bus 5004 may include a pathway for transferring information between various components of the device 5000 (eg, memory 5001, processor 5002, communication interface 5003).

It should be noted that although the device 3000 shown in FIG. 12 and the device 5000 shown in FIG. 14 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 3000, the device The 5000 also includes other devices necessary for proper operation. Meanwhile, according to specific needs, those skilled in the art should understand that the apparatus 3000 and the apparatus 5000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 3000 and the device 5000 may only include the devices necessary to realize the embodiment of the present application, instead of all the devices shown in FIG. 12 and FIG. 14 .

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. A skilled artisan may use different means to implement the described functions for each particular application, but such implementation should not be considered as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: Universal Serial Bus flash disk (UFD), UFD can also be referred to as U disk or USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., which can store program codes. medium.

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (25)

A line of sight detection method, characterized in that, comprising: Acquiring an image to be processed, where the image to be processed includes the eye region of the target object; Minimizing the difference between the estimated line of sight image and the image to be processed to obtain the line of sight of the target object, the estimated line of sight image is obtained by covering the eye area part with an eye area model, and the eye area model is obtained by using a three-dimensional The eyeball model is obtained by using the texture image of the eyeball to render the eyeball mesh. The method according to claim 1, wherein the eyeball grid is obtained using a three-dimensional human head model. The method according to claim 2, wherein the eyeball model is obtained by rendering the eyeball grid according to the mapping relationship between the eyeball grid and the texture image. The method of claim 3, wherein the mapping relationship is non-linear. The method according to any one of claims 1 to 4, wherein the difference between the estimated line of sight image and the image to be processed is represented by an energy function, and the minimized line of sight estimated image and the image to be processed Process image differences to obtain the line of sight of the target object, including: The energy function is minimized to obtain attitude parameters, and the attitude parameters include the line of sight. The method according to claim 5, wherein at least one of the following parameters is obtained when minimizing the energy function: face shape parameters, texture parameters or illumination parameters. A method for modeling an eyeball model, comprising: Obtain the texture image of the eye mesh and eyeball; The eyeball grid is rendered by using the texture image to obtain an eyeball model. The method according to claim 7, wherein the eyeball grid is obtained using a three-dimensional human head model. The method according to claim 7 or 8, wherein said rendering the eye grid using said texture image comprises: Establish a mapping relationship between the eye grid and the texture image; Render the texture of the texture image onto the eye grid according to the mapping relationship. The method of claim 9, wherein the mapping relationship is non-linear. The method according to any one of claims 8 to 10, further comprising: Putting the eyeball model into the human head model, the texture of the human head model is in a different quadrant from the texture of the eyeball model. A line-of-sight detection device, characterized in that it comprises: an acquisition unit, configured to acquire an image to be processed, the image to be processed includes the eye region of the target object; A processing unit, configured to minimize the difference between the estimated line of sight image and the image to be processed to obtain the line of sight of the target object, the estimated line of sight image is obtained by covering the eye area part with an eye area model, and the eye area The model is obtained by using a three-dimensional eyeball model, and the eyeball model is obtained by rendering an eyeball mesh using a texture image of the eyeball. The device according to claim 12, wherein said eyeball grid is obtained using a three-dimensional human head model. The device according to claim 13, wherein the eyeball model is obtained by rendering the eyeball grid according to the mapping relationship between the eyeball grid and the texture image. The apparatus of claim 14, wherein the mapping relationship is non-linear. The device according to any one of claims 12 to 15, wherein the difference between the line of sight estimation image and the image to be processed is represented by an energy function, and the processing unit is specifically used for: The energy function is minimized to obtain attitude parameters, and the attitude parameters include the line of sight. The device according to claim 16, wherein when the processing unit is used to minimize the energy function, at least one of the following parameters is obtained: a face shape parameter, a texture parameter or an illumination parameter. A modeling device for an eyeball model, characterized in that it comprises: An acquisition unit, configured to acquire texture images of eyeball grids and eyeballs; The processing unit is configured to use the texture image to render the eyeball grid to obtain an eyeball model. The device according to claim 18, wherein the eyeball grid is obtained by using a three-dimensional human head model. The device according to claim 18 or 19, wherein the processing unit is specifically used for: Establish a mapping relationship between the eye grid and the texture image; Render the texture of the texture image onto the eye grid according to the mapping relationship. The apparatus of claim 20, wherein the mapping relationship is non-linear. The device according to any one of claims 19 to 21, wherein the processing unit is further configured to: Putting the eyeball model into the human head model, the texture of the human head model is in a different quadrant from the texture of the eyeball model. A computer-readable storage medium, characterized in that the computer-readable medium stores program code for execution by a device, and the program code includes a program code for executing any one of claims 1 to 6 or claims 7 to 11. Instructions for the method. A computing device, characterized in that the device comprises a processor and a data interface, and the processor reads the instructions stored on the memory through the data interface to execute claims 1 to 6 or claims 7 to 11 any one of the methods described. A computer program product, characterized in that, when the computer program is executed on a computer, the computer is made to execute the method according to any one of claims 1-6 or 7-11.

Images