WO2022141114A1 - Line-of-sight estimation method and apparatus, vehicle, and computer-readable storage medium - Google Patents

Abstract

The present application provides a line-of-sight estimation method and apparatus, a vehicle, and a computer-readable storage medium. In embodiments of the present application, because a plurality of set areas are preset, and a process of secondary estimation is designed using two types of neural networks having different tasks, a first neural network first recognizes a target area that a user is facing from a large range, and then a second neural network corresponding to the target area is utilized to recognize target line-of-sight direction information of the user's line of sight in the target area. Hence, because the first neural network is first utilized to narrow the estimation range into the target area, and then the second neural network adapted to the target area is utilized to estimate the target line-of-sight direction information in the target area, the estimation result can have higher accuracy.

Description

Line-of-sight estimation method, apparatus, vehicle, and computer-readable storage medium technical field

The present application relates to the technical field of intelligent driving, and in particular, to a line of sight estimation method, device, vehicle, and computer-readable storage medium.

Background technique

Gaze is a behavior that reflects the user's attention. In the prior art, there are few methods of applying the gaze estimation method to intelligent driving scenarios and performing targeted optimization. How to accurately estimate the user's gaze direction is a technical problem that needs to be solved urgently.

SUMMARY OF THE INVENTION

In view of this, the present application provides a line-of-sight estimation method, apparatus, device, and computer-readable storage medium to solve the technical problem in the related art that the user's line-of-sight direction cannot be accurately estimated.

In a first aspect, a line-of-sight estimation method is provided, the method comprising:

Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user;

The target area corresponding to the image to be recognized is estimated by using the first neural network, the target area is at least one of a plurality of set areas, and the target area represents the area in the image to be recognized that the user's head is facing ;

A second neural network corresponding to the target area is acquired, and the target gaze direction information of the user is estimated by using the second neural network.

In a second aspect, a line-of-sight estimation method is provided, the method comprising:

Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user;

Use a neural network to estimate the user's target line of sight direction information; wherein, the neural network is used to: after extracting at least one type of target features representing the orientation of the user's head in the to-be-recognized image, fuse the extracted target features, The target gaze direction information of the user is estimated according to the feature fusion result.

In a third aspect, a line-of-sight estimation device is provided, the device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, the processor implements the following steps when executing the computer program :

Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user;

The target area corresponding to the image to be recognized is estimated by using the first neural network, the target area is at least one of a plurality of set areas, and the target area represents the area in the image to be recognized that the user's head is facing ;

A second neural network corresponding to the target area is acquired, and the target gaze direction information of the user is estimated by using the second neural network.

In a fourth aspect, a line-of-sight estimation device is provided, the device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, the processor implements the following steps when executing the computer program :

Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user;

Use a neural network to estimate the user's target line of sight direction information; wherein, the neural network is used to: after extracting at least one type of target features representing the orientation of the user's head in the to-be-recognized image, fuse the extracted target features, The target gaze direction information of the user is estimated according to the feature fusion result.

In a fifth aspect, a vehicle is provided, the vehicle comprising the device of the aforementioned third aspect or the aforementioned device of the aforementioned fourth aspect.

In a sixth aspect, a computer-readable storage medium is provided, where several computer instructions are stored thereon, and when the computer instructions are executed, the line-of-sight estimation method described in the foregoing first aspect is implemented.

In a seventh aspect, a computer-readable storage medium is provided, where several computer instructions are stored thereon, and when the computer instructions are executed, the line-of-sight estimation method described in the foregoing second aspect is implemented.

Applying the solution provided by this application, since multiple setting areas are preset, and the secondary estimation process is designed by using two types of neural networks with different tasks, the first neural network first identifies the direction the user is facing from a large range. target area, and then use the second neural network corresponding to the target area to identify the target sight line direction information of the user's sight line in the target area. It can be seen that since the first neural network is used to first narrow the estimation range to the target area, and then the second neural network adapted to the target area is used to estimate the target line-of-sight direction information in the target area, so that the estimation results can be obtained relatively high accuracy.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

FIG. 1A is a schematic diagram of a line-of-sight estimation method according to an embodiment of the present application.

FIG. 1B is a schematic diagram of a line-of-sight estimation method according to another embodiment of the present application.

FIG. 1C is a schematic structural diagram of a first neural network/second neural network according to an embodiment of the present application.

FIG. 1D is a schematic diagram of a line-of-sight estimation method according to another embodiment of the present application.

FIG. 1E is a schematic diagram of a line-of-sight estimation method according to another embodiment of the present application.

FIG. 2 is a schematic diagram of a line-of-sight estimation method according to another embodiment of the present application.

FIG. 3 is a schematic structural diagram of an apparatus for implementing the line-of-sight estimation method of this embodiment in the present application.

FIG. 4 is a schematic diagram of a vehicle according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments.

With the development of data and technology, since the sight line information can reflect the user's attention, there are more and more concerns and demands on the user's sight line. It is necessary to provide a technical solution that can accurately estimate the user's sight line.

Most of the line-of-sight estimation schemes are implemented by machine learning. In the field of machine learning, a model is usually first expressed by modeling, then the model is evaluated by constructing an evaluation function, and finally the evaluation function is optimized according to sample data and optimization methods. , to adjust the model to the optimum. The whole stage involves a lot of links, such as the selection and processing of sample data, the design of data features, the design of models, the design of loss functions or the design of optimization methods, etc. The subtle differences in any link lead to subtle estimation accuracy. defect factor.

The inventor of the present application has found that one goal of line of sight estimation is to estimate line-of-sight direction information, that is, the direction of the user's line of sight; when the user pays attention to a certain position, the user's eyes are directed toward the position, thereby generating line-of-sight direction information, that is, the direction of the user's line of sight. The orientation will correspond to the position the user is focusing on. The user pays attention to other positions, and the corresponding user's line of sight faces other positions, and the line-of-sight direction information corresponding to the other positions is generated. The user's eyes cover a large range of sight. In a space, the user's eyes can be directed to any position. The target to be estimated (that is, the line-of-sight direction information corresponding to the specific position the user's eyes are facing) is relative to the entire line of sight covered by the line of sight. The range is very small, so it is difficult for the machine model to estimate the line-of-sight direction information of the user's line of sight from a large coverage area, and the result of line-of-sight estimation is difficult to achieve very high accuracy.

The inventor of the present application also found that the existing line-of-sight estimation methods are rarely optimized for intelligent driving scenarios. The traditional line-of-sight estimation method is based on image processing technology, which has poor robustness and cannot adapt to changes in illumination; With the development of learning-related technologies, some methods have been used to estimate sight lines through neural networks, but the shortcomings of the existing technologies are: failure to fully utilize facial information and status; failure to consider timing information; failure to match intelligent driving scenarios Combined, the accuracy of line-of-sight estimation cannot be improved by certain prior knowledge. The above deficiencies result in that the line of sight estimation cannot accurately estimate the driver's line of sight in an intelligent driving scenario, and it is difficult to make a correct judgment on whether the driver is distracted driving.

Therefore, the embodiment of the present application proposes a line-of-sight estimation scheme, which pre-sets multiple setting regions, and uses two types of neural networks with different tasks to design a secondary estimation process. Identify the target area that the user is facing, and then use the second neural network corresponding to the target area to identify the target line of sight direction information of the user's line of sight in the target area. It can be seen that since the first neural network is used to first narrow the estimation range to the target area, and then the second neural network adapted to the target area is used to estimate the target line-of-sight direction information in the target area, so that the estimation results can be obtained relatively high accuracy. Through a priori region discrimination and multi-information fusion, in the application scenario of intelligent driving, the line of sight can be estimated more accurately, which further helps the driver to judge whether the driver is distracted driving and improves driving safety. Next, the solution of this embodiment will be described in detail.

As shown in FIG. 1A , it is a flowchart of a line-of-sight estimation method according to an exemplary embodiment of the present application, including the following steps:

In step 102, at least one frame of to-be-recognized image is acquired, and the to-be-recognized image includes an image obtained by performing image acquisition on a user;

In step 104, a first neural network is used to estimate a target area corresponding to the to-be-recognized image, the target area is at least one of a plurality of set areas, and the target area represents the user's head in the to-be-recognized image the area towards which the part is directed;

In step 106, a second neural network corresponding to the target area is acquired, and the target gaze direction information of the user is estimated by using the second neural network.

The description will be made with reference to the schematic diagram of another line-of-sight estimation method shown in FIG. 1B . In this embodiment, it is considered that the user's line of sight will cover a large area in the actual scene, and the goal of line-of-sight estimation is to accurately identify when the user's eyes are looking at a specific position in the entire covered area. Line-of-sight information. Based on this, this embodiment pre-configures multiple setting areas, which are the targets of line of sight estimation. The configuration of these setting areas can be flexibly configured according to the needs of actual application scenarios. In FIG. 1B, the setting areas are i. The setting area j and the setting area k are taken as an example for illustration, which is not limited in this embodiment.

One application scenario of line of sight estimation is an assisted driving scenario, in which the result of line of sight estimation can be used to detect whether the driver is distracted driving. As an example, the multiple set areas include: front windshield area, left rearview mirror area, right mirror area, left window area, right window area, instrument panel area or center console area.

Other application scenarios of gaze estimation can be game scenarios, such as using the results of gaze estimation for game interaction; medical scenarios, such as using eye trackers to estimate the user's gaze direction; offline retail scenarios, such as using The line-of-sight estimation result predicts the user's degree of attention to the sold item, etc. In different application scenarios, the user's line-of-sight coverage can be divided into multiple set areas as required, which is not limited in this embodiment.

In this embodiment, the image to be recognized refers to an image obtained by capturing an image of a user. As an example, the image capturing manner may be obtained by capturing a user by a camera device.

In this embodiment, the task of the first neural network is to distinguish the area in which the user's head is facing in the image to be recognized, and the task of the second neural network is to estimate the target gaze direction information of the user. In some examples, each set area may correspond to a second neural network; in other examples, the set area and the second neural network may also have a many-to-one relationship, that is, one neural network may be applicable to two one or more setting fields. For convenience of illustration, FIG. 1B shows the setting area and the second neural network in a one-to-one manner, which may be configured as required in practical applications, which is not limited in this embodiment.

Based on the task of the first neural network, in the training image set used to train the first neural network, the area facing the user's head in each training image covers the multiple set areas, so that the trained first neural network can accurately Identify the area that the user's head is facing in the image; for example, if the setting area includes: setting area A, setting area B and setting area C, the training data of the first neural network includes: the user's line of sight direction The information belongs to the training image of the set area A, the user's gaze direction information belongs to the training image of the set area B, and the user's gaze direction information belongs to the training image of the set area C. Based on this, the first neural network can be trained, and the first neural network is used to estimate the target area corresponding to the image to be recognized. The estimation of orientation information is narrowed from a large range to the range of the target area. Since the first neural network can identify a small area in a large range without accurately identifying specific line-of-sight direction information, the output result of the first neural network has high reliability.

Based on the task of the second neural network, the training image set used to train the second neural network corresponding to the set area adopts the image of the area facing the user's head as the set area; for example, the set area includes : Setting area A, setting area B, and setting area C, the training data of the second neural network corresponding to setting area A is the training image whose line of sight direction information of the user belongs to setting area A, and setting area B corresponds to The training data of the second neural network is the training image of the user's line of sight direction information belonging to the set area B, and the training data of the second neural network corresponding to the set area C is the user's line of sight direction information belongs to the set area C. training images. Therefore, the second neural network corresponding to the set area focuses on the line-of-sight feature of the user's line-of-sight direction of the set area, so that the second neural network can accurately identify the line-of-sight direction information. Here, the second neural network is adapted to the target area, and the second neural network identifies specific line-of-sight direction information in a smaller target area, so the output of the second neural network also has high reliability. Finally, the line-of-sight estimation results have higher accuracy.

Optionally, the training process can be supervised. The aforementioned training images can be calibrated with user gaze direction information, and the calibrated user gaze direction information will correspond to one of the multiple setting areas, and the training images are also calibrated. its corresponding setting area. In some examples, the training images may be virtual datasets, ie, images obtained in a simulation manner, which can efficiently obtain training images. In other examples, the training images can be real datasets, because the real-world disturbances will be mixed in the images during the actual collection of user images, and these disturbances will make the neural network have better performance during the training of the neural network. Therefore, it can output more accurate estimation results when dealing with the actual environment in the estimation stage.

In this embodiment, the input data of the first neural network and the second neural network are images to be recognized. As an example, the first neural network is used to estimate the target area corresponding to the image to be recognized, and the estimation process may be: Extracting image features of the to-be-recognized image, and using the image features to estimate a target area corresponding to the to-be-recognized image. The second neural network is used for estimating the line of sight direction information of the image to be recognized, and the estimation processing process may be: extracting image features of the to-be-recognized image, and using the image features to estimate the line of sight direction information of the user.

In some examples, when designing the first neural network and the second neural network, different network structures may be used, and the specific network structure design may be flexibly configured as required. In other examples, the estimation process of the first neural network and the second neural network are the same as a whole, but the output results are different. In order to improve the efficiency, the structure of the first neural network is the same as that of the second neural model. That is, the two can adopt the same network structure design. Optionally, because the output results of the two are different, the parameters of the first neural network obtained by training are different from the parameters of the second neural model. Since the input of the first neural network and the second neural network is an image, optionally, a convolutional neural network can be well suited for image processing, the first neural network can be a convolutional neural network, and the second neural network can also be Convolutional Neural Networks. Of course, other forms of the first neural network and the second neural network can also be flexibly designed in practical applications, which are not limited in this embodiment.

Based on the overall consistent estimation process of the first neural network and the second neural network, for the first neural network/second neural network, image features are identified from the input image to be recognized, and image features are used to estimate the image to be recognized. Corresponding target area/target line-of-sight direction information. Considering the goal of line-of-sight estimation, in order to enable the first neural network and the second neural network to output accurate results, the first neural network/second neural network is also innovatively designed in this embodiment:

In some examples, the image features include any of the following: face region features, eye region features, or head pose features, considering the target of gaze estimation. In some examples, the face region feature, eye region feature or head pose feature are correspondingly extracted from the face region, eye region or head region segmented from the to-be-recognized image.

In some examples, the eye region features include any one of the following: binocular features, left eye features or right eye features.

In some examples, the left-eye feature and the right-eye feature are extracted by using the Siamese network. In some examples, the Siamese network is used to perform dilated convolution operations on the left eye region and the right eye region respectively, so as to enhance the network's receptive field of the image.

As shown in FIG. 1C , it is a schematic diagram of the first neural network/second neural network structure shown in this embodiment, and the network structure in FIG. 1C is applicable to the first neural network and the second neural network. Among them, INPUT is the image to be recognized, and the network structure includes an image segmentation layer face detection, which is used to divide the image to be recognized into three parts: face area, eye area landmark and head area headpose.

For the face region, to identify the features of the face region, the key points of the face can be used, such as 68 key points of the face; the features of the entire face can also be extracted through the face region feature extraction layer CNN.

For the eye region, the binocular region pitch, the left eye region and the right eye region can be segmented through the segmentation operation crop. For the binocular area pitch, the binocular features can be extracted through the binocular feature extraction layer. For the left eye area and the right eye area, the twin network layer can be used to extract the left eye feature and the right eye feature from the left eye area and the right eye area. The twin network can share parameters based on the left eye area and the right eye area. The difference and connection extraction features. Optionally, the Siamese network can also be used to perform dilated convolution operations on the left-eye region and the right-eye region respectively, so as to enhance the network's receptive field of the image, thereby extracting accurate left-eye and right-eye features.

For the head region headpose, head pose features can be identified.

All the extracted features can be fused through the fully connected layer FC, and then output the estimation result; for the first neural network, the output estimation result is the target area gaze zone; for the second neural network, the output estimation is the target line of sight direction information , specifically the angle information gaze pitch&yaw of the line of sight direction.

In addition to the embodiment shown in FIG. 1C , the first neural network can estimate the target area corresponding to the image to be recognized by recognizing the user's gaze direction information in the image to be recognized. In other embodiments, the first neural network can also be implemented in other ways, for example, it can be a neural network with other structures. In order to improve efficiency, a simple-structured neural network can also be used to recognize the user's identity from the input image to be recognized. The head pose information is used to estimate the target area. This method does not need to consider other information such as eyes and key points. A neural network with a relatively simple structure is used, which can reduce the implementation cost.

In some examples, the number of frames of the to-be-recognized image may be one frame, and after the to-be-recognized image of the frame is input to the first neural network, the first neural network identifies the target area corresponding to the to-be-recognized image; The second neural network corresponding to the target area inputs the to-be-recognized image of the frame into the second neural network, and uses the second neural network to estimate the target line-of-sight direction information of the user.

In other examples, there may be more than one frame of the image to be recognized, and sight line estimation can be performed for multiple frames of the image to be recognized. The specific number of frames can be flexibly configured as required, for example, a fixed number of frames can be set, such as 3 frames, 5 frame etc. In other examples, the number of image frames for each line-of-sight estimation can be dynamically changed, and the number of image frames can also be determined by setting the time interval for line-of-sight estimation. It can be estimated by taking one or more frames of images in this time period.

As an example, the multiple frames of images to be identified include: multiple frames of images photographed by the same photographing device on the user at different times; or multiple frames of images photographed by different photographing devices on the user at different locations. For the line-of-sight estimation scheme of the multi-frame to-be-recognized images, there can be implemented in various ways.

As an example, it is possible to estimate multiple line-of-sight direction information for multiple frames of images to be identified and then fuse; as shown in FIG. 1D , the processing process for multiple frames of images to be identified is shown. (The n-1th frame, the nth frame, and the n+1th frame) are examples to illustrate an embodiment of the line of sight estimation. For each frame of the image to be identified, the corresponding target area can be identified by the first neural network, and then Each frame of the to-be-recognized image is input to the corresponding second neural network. As an example, using the second neural network to estimate the user's target gaze direction information includes: acquiring the user's gaze direction information in each frame of the image to be recognized; the user's gaze direction information in each frame of the to-be-recognized image is The second neural network corresponding to the target area corresponding to the frame to be recognized image is obtained by estimating the frame to be recognized image; the user's target gaze direction information is estimated after fusing the line of sight direction information corresponding to each frame of the to-be-recognized image. The multiple frames of images to be identified shown in the schematic diagram of FIG. 1D all correspond to the same target area. In practical applications, there may also be situations where multiple frames of images to be identified correspond to different target areas, which is not limited in this embodiment.

In other examples, for multiple frames of images to be identified, after the image features are extracted respectively, the image features are fused, and then a plurality of line-of-sight direction information is estimated according to the feature fusion result. Specifically, using the second neural network to estimate the target line of sight direction information of the user includes: acquiring the image features of each frame of the image to be recognized; the image features of each frame of the image to be recognized correspond to the image to be recognized The second neural network corresponding to the target area of the frame is extracted from the image to be recognized in the frame; different from the previous embodiment, this embodiment estimates the user's target line of sight direction information after fusing the image features of each frame of the image to be recognized. Optionally, in order to further improve the accuracy of the estimation result, the third neural network may be used to estimate the user's target line-of-sight direction information after fusing the image features of each frame of the image to be recognized.

The line-of-sight estimation solution in this embodiment can be applied to various business scenarios, such as assisted driving, games, VR (Virtual Reality, virtual reality), or medical treatment.

Next, taking an assisted driving scenario as an example, the line-of-sight estimation solution of this embodiment will be described. In the assisted driving scene, the to-be-identified image may include: an image taken by the driver in the vehicle car; based on the assisted driving scene, a plurality of set areas may be divided according to the driver's line of sight coverage in the vehicle car, As an example, the plurality of setting areas may include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, a dashboard area or a center console area; of course, In practical applications, other forms of setting areas may be divided according to requirements, which are not limited in this embodiment. In some examples, other regions can also be divided as required. These other regions do not correspond to the second neural network, and are not the estimation targets of the line-of-sight estimation scheme. When the first neural network identifies that the image to be recognized corresponds to other regions , you can output the recognition result, and end the line of sight estimation process.

As shown in FIG. 1E, it is an embodiment of the line of sight estimation method in the assisted driving scene shown in this embodiment. The image to be recognized as an INPUT is input into the first neural network CNN, and the image to be recognized is output by the first neural network CNN. The corresponding target area gaze zone; the setting area divided in this embodiment is shown in Figure 1E, and each setting area corresponds to a second neural network, namely CNN-1, CNN-2 to CNN-7 in Figure 1E, According to the first neural network CNN, the target area gaze zone corresponding to the image to be recognized is output, the corresponding second neural network is obtained, the image to be recognized is input to the second neural network, and the gaze direction information gaze is output by the second neural network.

Optionally, the estimated target line-of-sight direction information may be used to assess whether the driver is distracted driving. As an example, the target line of sight direction information may be used to determine that the user's line of sight direction deviates from the set direction, and prompt information is output to prompt the driver. Optionally, the output mode of the prompt information includes any of the following modes: audio, text, video or lighting. For example, the prompt message can be played through audio, or the prompt message can be output in the form of text or video through the electronic screen in the car, or the light module in the car can be controlled to output the prompt message in a specific light color. The combination of the above at least two ways outputs a prompt message.

In the solution of this embodiment, the first neural network is used to estimate the target area corresponding to the to-be-recognized image, the target area is at least one of a plurality of set areas, and the target area represents the location of the user's head in the to-be-recognized image towards the area. The set area corresponds to a second network, and the second neural network is used to estimate the target line of sight direction information of the user, so that on the basis of identifying the current area by the first neural network, the second neural network is used to perform more detailed and accurate line-of-sight estimate.

The embodiment of this specification also provides another line-of-sight estimation method. The method of this embodiment is aimed at a single neural network. As mentioned above, this embodiment also achieves innovation on a single neural network, as shown in FIG. 2 . , is a schematic flowchart of the line-of-sight estimation method, including the following steps:

In step 202, at least one frame of images to be recognized is acquired. The to-be-recognized image includes an image obtained by performing image acquisition on a user;

In step 204, the target gaze direction information of the user is estimated by using a neural network. The neural network is used for: after extracting at least one type of target features representing the orientation of the user's head in the to-be-recognized image, fuse the extracted target features, and estimate the user's target line-of-sight direction information according to the feature fusion result .

In this embodiment, the neural network is used to extract at least one type of target features representing the orientation of the user's head in the image to be recognized, and the extracted target features are fused, so that the target gaze direction of the user can be accurately estimated according to the feature fusion result. information.

In some examples, the target features include any of the following: face region features, eye region features, or head pose features.

In some examples, the face region feature, eye region feature or head pose feature are correspondingly extracted from the face region, eye region or head region segmented from the to-be-recognized image.

In some examples, the eye region features include any one of the following: binocular features, left eye features or right eye features.

In some examples, the left-eye feature and the right-eye feature are extracted by using the Siamese network.

In some examples, the Siamese network is used to perform dilated convolution operations on the left eye region and the right eye region respectively, so as to enhance the network's receptive field of the image.

In some examples, the target line-of-sight direction information of the user includes: a target area, where the target area represents an area in the to-be-recognized image to which the user's head faces.

In some examples, the training data of the neural network adopts an image in which the region facing the user's head is the target region.

In some examples, the target gaze direction information of the user includes: angle information of the user's target gaze direction.

In some examples, the to-be-identified image has multiple frames;

The using the neural network to estimate the target gaze direction information of the user includes:

Obtaining the line-of-sight direction information of the user in each frame of the to-be-recognized image; the user's line-of-sight direction information in each frame of the to-be-recognized image is estimated by the neural network corresponding to the to-be-recognized image of the frame of the to-be-recognized image;

The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information.

In some examples, the to-be-recognized image has multiple frames; the neural network includes an initial sub-neural network and a fusion sub-neural network;

The using the neural network to estimate the target gaze direction information of the user includes:

For each frame of the to-be-recognized image, the initial sub-neural network is used to obtain the feature fusion result of each frame of the to-be-recognized image; the initial sub-neural network is used to: extract at least one type of target representing the user's head orientation in the to-be-recognized image After the feature, the extracted target features are fused;

Using the fusion sub-neural network, the feature fusion results of each frame of the image to be recognized are fused to estimate the user's target line-of-sight direction information.

In some examples, the neural network is a convolutional neural network.

In some examples, the initial sub-neural network is a convolutional neural network.

In some examples, the fusion sub-neural network is a recurrent neural network.

In some examples, the image to be identified includes an image taken of a driver in the vehicle.

In some examples, the target gaze direction information is used to assess whether the driver is distracted driving.

In some examples, the plurality of setting areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, a dashboard area, or a center console area .

In some examples, the method further includes:

Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output.

In some examples, the output mode of the prompt information includes any one of the following modes: audio, text, video or lighting.

For the implementation process of the line-of-sight estimation method in this embodiment, reference may be made to the foregoing descriptions of the embodiments shown in FIG. 1A to FIG. 1E , which will not be repeated here.

The foregoing method embodiments may be implemented by software, and may also be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory through the processor that estimates the line of sight where it is located. From the perspective of hardware, as shown in FIG. 3 , which is a hardware structure diagram for implementing the line-of-sight estimation apparatus 300 in this embodiment, except for the processor 301 and the memory 302 shown in FIG. The processing device of the line-of-sight estimation method, usually according to the actual function of the processing device, may also include other hardware, which will not be repeated here.

In this embodiment, the line-of-sight estimation apparatus 300 includes a processor, a memory, and a computer program stored on the memory and executable by the processor, and the processor implements the following steps when executing the computer program:

Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user;

The target area corresponding to the image to be recognized is estimated by using the first neural network, the target area is at least one of a plurality of set areas, and the target area represents the area in the image to be recognized that the user's head is facing ;

A second neural network corresponding to the target area is acquired, and the target gaze direction information of the user is estimated by using the second neural network.

In some examples, the training image set used for training the second neural network corresponding to the set area adopts the images of the area to which the user's head is facing is the set area.

In some examples, in the training image set used for training the first neural network, the area to which the user's head faces in each training image covers the multiple set areas.

In some examples, the to-be-identified image has multiple frames;

When the processor executes the estimation of the target gaze direction information of the user by using the second neural network, the following steps are implemented:

Obtain the user's gaze direction information in each frame of the image to be recognized; the user's gaze direction information in each frame of the to-be-recognized image is estimated by the second neural network corresponding to the target area corresponding to the to-be-recognized image for the frame to be recognized. of;

The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information.

In some examples, the second neural network is used for: extracting image features of the to-be-recognized image, and estimating the user's gaze direction information by using the image features.

In some examples, the to-be-identified image has multiple frames;

When the processor executes the estimation of the target gaze direction information of the user by using the second neural network, the following steps are implemented:

Obtaining the image features of each frame of the image to be recognized; the image feature of each frame of the image to be recognized is extracted from the image to be recognized by the second neural network corresponding to the target area corresponding to the image to be recognized in the frame;

The target gaze direction information of the user is estimated after the image features of each frame of the image to be recognized are fused.

In some examples, the processor implements the following steps when estimating the target gaze direction information of the user after merging the image features of each frame of the image to be recognized:

Using the third neural network to fuse the image features of the images to be identified in each frame, the user's target line-of-sight direction information is estimated.

In some examples, the first neural network is used to: extract image features of the to-be-recognized image, and use the image features to estimate a target area corresponding to the to-be-recognized image.

In some examples, the image features include any of the following: face region features, eye region features, or head pose features.

In some examples, the face region feature, eye region feature or head pose feature are correspondingly extracted from the face region, eye region or head region segmented from the to-be-recognized image.

In some examples, the eye region features include any one of the following: binocular features, left eye features or right eye features.

In some examples, the left-eye feature and the right-eye feature are extracted by using the Siamese network.

In some examples, the Siamese network is used to perform atrous convolution operations on the left eye region and the right eye region respectively, so as to enhance the network's receptive field of the image.

In some examples, the multiple frames of images to be identified include: multiple frames of images captured by the same photographing device on the user at different times; or multiple frames of images photographed by different photographing devices on the user at different locations.

In some examples, the target area is estimated by the first neural network by recognizing the user's gaze direction information and/or the user's head posture information in the to-be-recognized image.

In some examples, the structure of the first neural network is the same as the structure of the second neural model.

In some examples, the parameters of the first neural network are different from the parameters of the second neural model.

In some examples, the first neural network is a convolutional neural network and/or the second neural network is a convolutional neural network.

In some examples, the third neural network is a recurrent neural network.

In some examples, the target area is one of a plurality of set areas.

In some examples, the to-be-identified image includes an image taken of a driver in a vehicle cabin.

In some examples, the target gaze direction information is used to assess whether the driver is distracted driving.

In some examples, the plurality of set areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, a dashboard area, or a center console area .

In some instances, the processor further implements the following steps when executing the computer program:

Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output.

In some examples, the output mode of the prompt information includes any one of the following modes: audio, text, video or lighting.

This embodiment also provides another apparatus for line-of-sight estimation. The structure of the apparatus may be as shown in FIG. 3 . The apparatus includes a processor, a memory, and a computer program stored in the memory and executed by the processor. The processor implements the following steps when executing the computer program:

Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user;

A neural network is used to estimate the target line of sight direction information of the user; wherein, the neural network is used to: after extracting at least one type of target features representing the orientation of the user's head in the to-be-recognized image, fuse the extracted target features, The target gaze direction information of the user is estimated according to the feature fusion result.

In some examples, the target features include any of the following: face region features, eye region features, or head pose features.

In some examples, the face region feature, eye region feature or head pose feature are correspondingly extracted from the face region, eye region or head region segmented from the to-be-recognized image.

In some examples, the eye region features include any one of the following: binocular features, left eye features or right eye features.

In some examples, the left-eye feature and the right-eye feature are extracted by using the Siamese network.

In some examples, the Siamese network is used to perform dilated convolution operations on the left eye region and the right eye region respectively, so as to enhance the network's receptive field of the image.

In some examples, the target line-of-sight direction information of the user includes: a target area, where the target area represents an area in the to-be-recognized image to which the user's head faces.

In some examples, the training data of the neural network adopts an image in which the region facing the user's head is the target region.

In some examples, the target gaze direction information of the user includes: angle information of the user's target gaze direction.

In some examples, the to-be-identified image has multiple frames;

When the processor executes the estimation of the target gaze direction information of the user by using the neural network, the following steps are implemented:

Obtaining the line-of-sight direction information of the user in each frame of the to-be-recognized image; the user's line-of-sight direction information in each frame of the to-be-recognized image is estimated by the neural network corresponding to the to-be-recognized image of the frame of the to-be-recognized image;

The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information.

In some examples, the to-be-recognized image has multiple frames; the neural network includes an initial sub-neural network and a fusion sub-neural network;

When the processor executes the estimation of the target gaze direction information of the user by using the neural network, the following steps are implemented:

For each frame of the to-be-recognized image, the initial sub-neural network is used to obtain the feature fusion result of each frame of the to-be-recognized image; the initial sub-neural network is used for: extracting at least one type of target representing the orientation of the user's head in the to-be-recognized image After the feature, the extracted target features are fused;

Using the fusion sub-neural network, the feature fusion results of each frame of the image to be recognized are fused to estimate the user's target line-of-sight direction information.

In some examples, the neural network is a convolutional neural network.

In some examples, the initial sub-neural network is a convolutional neural network.

In some examples, the fusion sub-neural network is a recurrent neural network.

In some examples, the image to be identified includes an image taken of a driver in the vehicle.

In some examples, the target gaze direction information is used to assess whether the driver is distracted driving.

In some examples, the plurality of set areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, a dashboard area, or a center console area .

In some instances, the processor further implements the following steps when executing the computer program:

Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output.

In some examples, the output mode of the prompt information includes any one of the following modes: audio, text, video or lighting.

As shown in FIG. 4 , an embodiment of the application further provides a vehicle 400 , and the vehicle may include any of the aforementioned line-of-sight estimation devices.

This embodiment further provides a computer-readable storage medium, where several computer instructions are stored thereon, and when the computer instructions are executed, the steps of the line-of-sight estimation method shown in FIG. 1A are implemented.

This embodiment also provides a computer-readable storage medium, characterized in that, the readable storage medium stores several computer instructions, and when the computer instructions are executed, the steps of the line-of-sight estimation method shown in FIG. 2 are implemented.

Embodiments of the present specification may take the form of a computer program product embodied on one or more storage media having program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like. Computer-usable storage media includes permanent and non-permanent, removable and non-removable media, and storage of information can be accomplished by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

For the apparatus embodiments, since they basically correspond to the method embodiments, reference may be made to the partial descriptions of the method embodiments for related parts. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also other not expressly listed elements, or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The methods and devices provided by the embodiments of the present invention have been described in detail above. The principles and implementations of the present invention are described in this paper by using specific examples. At the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this description should not be construed as a limitation of the present invention. .

Claims (92)

A line-of-sight estimation method comprising: Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user; The target area corresponding to the image to be recognized is estimated by using the first neural network, the target area is at least one of a plurality of set areas, and the target area represents the area in the image to be recognized that the user's head is facing ; A second neural network corresponding to the target area is acquired, and the target gaze direction information of the user is estimated by using the second neural network. The method according to claim 1, wherein the training image set used for training the second neural network corresponding to the set area adopts the image of the set area where the area facing the user's head is. The method according to claim 1, wherein, in the training image set used for training the first neural network, the area to which the user's head faces in each training image covers the plurality of set areas. The method according to claim 1, wherein the to-be-identified image has multiple frames; The using the second neural network to estimate the target gaze direction information of the user includes: Obtain the line of sight direction information of the user in each frame of the to-be-recognized image; the user's line-of-sight direction information in each frame of the to-be-recognized image is estimated by the second neural network corresponding to the target area corresponding to the to-be-recognized image of the frame of the to-be-recognized image. of; The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information. The method according to claim 1 or 4, wherein the second neural network is used for: extracting image features of the to-be-recognized image, and estimating the user's gaze direction information by using the image features. The method according to claim 1 or 5, wherein the to-be-identified image has multiple frames; The using the second neural network to estimate the target gaze direction information of the user includes: Obtaining the image features of each frame of the image to be recognized; the image feature of each frame of the image to be recognized is extracted from the image to be recognized by the second neural network corresponding to the target area corresponding to the image to be recognized in the frame; The target gaze direction information of the user is estimated after the image features of each frame of the image to be recognized are fused. The method according to claim 6, wherein the estimating the user's target line-of-sight direction information after fusing the image features of each frame of the image to be recognized comprises: The user's target line of sight direction information is estimated by using the third neural network to fuse the image features of each frame of the image to be recognized. The method according to claim 1, wherein the first neural network is used to: extract image features of the to-be-recognized image, and use the image features to estimate a target area corresponding to the to-be-recognized image. The method according to claim 6 or 8, wherein the image features include any one of the following: a face region feature, an eye region feature, or a head pose feature. The method according to claim 9, wherein the face region feature, the eye region feature or the head pose feature are respectively the face region, the eye region or the head segmented from the to-be-recognized image Correspondingly extracted from the partial area. The method according to claim 10, wherein the eye region feature includes any one of the following: a binocular feature, a left eye feature or a right eye feature. The method according to claim 11, wherein the left-eye feature and the right-eye feature are extracted by using a Siamese network. The method according to claim 12, wherein the twin network is used for: performing hole convolution operations on the left eye region and the right eye region respectively, so as to enhance the receptive field of the network to the image. The method according to claim 4 or 6, wherein the multiple frames of images to be identified comprise: multiple frames of images captured by the same shooting device at different times of the user; or, images captured by different shooting devices at different locations of the user multi-frame images. The method according to claim 1 or 8, wherein the target area is estimated by the first neural network by identifying the user's gaze direction information and/or the user's head posture information in the to-be-recognized image out. The method of claim 1, wherein the structure of the first neural network is the same as the structure of the second neural model. The method of claim 16, wherein the parameters of the first neural network are different from the parameters of the second neural model. The method of claim 16, wherein the first neural network is a convolutional neural network. The second neural network is a convolutional neural network. The method of claim 16, wherein the second neural network is a convolutional neural network. The method according to claim 7, wherein the third neural network is a recurrent neural network. The method of claim 1, wherein the target area is one of a plurality of setting areas. The method according to claim 1, wherein the to-be-identified image comprises: an image taken of a driver in a car of a vehicle. The method of claim 22, wherein the target gaze direction information is used to assess whether the driver is distracted driving. The method according to claim 22, wherein the plurality of setting areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, Dashboard area or center console area. The method according to claim 1, wherein the method further comprises: Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output. The method according to claim 25, wherein the output mode of the prompt information includes any one of the following modes: audio, text, video or lighting. A line-of-sight estimation method, characterized in that the method comprises: Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user; A neural network is used to estimate the target line of sight direction information of the user; wherein, the neural network is used to: after extracting at least one type of target features representing the orientation of the user's head in the to-be-recognized image, fuse the extracted target features, The target gaze direction information of the user is estimated according to the feature fusion result. The method according to claim 27, wherein the target feature comprises any one of the following: a face region feature, an eye region feature or a head pose feature. The method according to claim 28, wherein the face region feature, eye region feature or head pose feature are respectively a face region, an eye region or a head region segmented from the to-be-recognized image Correspondingly extracted from the partial area. The method according to claim 29, wherein the eye region feature comprises any one of the following: a binocular feature, a left eye feature or a right eye feature. The method according to claim 30, wherein the left-eye feature and the right-eye feature are extracted by using a Siamese network. The method according to claim 31, wherein the twin network is used for: performing hole convolution operations on the left eye region and the right eye region respectively, so as to enhance the receptive field of the network to the image. The method according to claim 27, wherein the target line-of-sight direction information of the user comprises: a target area, and the target area represents an area in the to-be-recognized image to which the user's head is facing. The method according to claim 27, wherein the training data of the neural network adopts the image of the target area in which the area to which the user's head is facing is used. The method according to claim 27, wherein the information on the target gaze direction of the user comprises: angle information of the user's target gaze direction. The method according to claim 27, wherein the to-be-identified image has multiple frames; The using the neural network to estimate the target gaze direction information of the user includes: Obtaining the line-of-sight direction information of the user in each frame of the to-be-recognized image; the user's line-of-sight direction information in each frame of the to-be-recognized image is estimated by the neural network corresponding to the to-be-recognized image of the frame of the to-be-recognized image; The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information. The method according to claim 27, wherein the image to be recognized has multiple frames; the neural network comprises an initial sub-neural network and a fusion sub-neural network; The using the neural network to estimate the target gaze direction information of the user includes: For each frame of the to-be-recognized image, the initial sub-neural network is used to obtain the feature fusion result of each frame of the to-be-recognized image; the initial sub-neural network is used for: extracting at least one type of target representing the orientation of the user's head in the to-be-recognized image After the feature, the extracted target features are fused; Using the fusion sub-neural network, the feature fusion results of each frame of the image to be recognized are fused to estimate the user's target line-of-sight direction information. The method of claim 36, wherein the neural network is a convolutional neural network. The method according to claim 37, wherein the initial sub-neural network is a convolutional neural network. The method of claim 37, wherein the fusion sub-neural network is a recurrent neural network. The method according to claim 27, wherein the to-be-identified image comprises: an image taken of a driver in the vehicle. The method of claim 41, wherein the target gaze direction information is used to assess whether the driver is distracted driving. The method according to claim 41, wherein the plurality of setting areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, Dashboard area or center console area. The method of claim 42, wherein the method further comprises: Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output. The method according to claim 44, wherein the output mode of the prompt information includes any one of the following modes: audio, text, video or lighting. A line-of-sight estimation device, characterized in that the device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, and the processor implements the following steps when executing the computer program: Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user; The target area corresponding to the image to be recognized is estimated by using the first neural network, the target area is at least one of a plurality of set areas, and the target area represents the area in the image to be recognized toward which the user's head is facing ; A second neural network corresponding to the target area is acquired, and the target gaze direction information of the user is estimated by using the second neural network. The device according to claim 46, wherein the training image set used for training the second neural network corresponding to the set area adopts the image of the set area where the area facing the user's head is. The apparatus according to claim 46, wherein, in the training image set used for training the first neural network, the area to which the user's head faces in each training image covers the plurality of set areas. The device according to claim 46, wherein the to-be-identified image has multiple frames; When the processor executes the estimation of the target gaze direction information of the user by using the second neural network, the following steps are implemented: Obtain the line of sight direction information of the user in each frame of the to-be-recognized image; the user's line-of-sight direction information in each frame of the to-be-recognized image is estimated by the second neural network corresponding to the target area corresponding to the to-be-recognized image of the frame of the to-be-recognized image. of; The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information. The apparatus according to claim 46 or 49, wherein the second neural network is used for: extracting image features of the to-be-recognized image, and estimating the user's gaze direction information by using the image features. The device according to claim 46 or 50, wherein the to-be-identified image has multiple frames; When the processor executes the estimation of the target gaze direction information of the user by using the second neural network, the following steps are implemented: Obtaining the image features of each frame of the image to be recognized; the image feature of each frame of the image to be recognized is extracted from the image to be recognized by the second neural network corresponding to the target area corresponding to the image to be recognized in the frame; After fusing the image features of the images to be recognized in each frame, the user's target gaze direction information is estimated. The device according to claim 51, wherein the processor implements the following steps when estimating the target line-of-sight direction information of the user after fusing the image features of each frame of the image to be recognized: The user's target line of sight direction information is estimated by using the third neural network to fuse the image features of each frame of the image to be recognized. The apparatus according to claim 46, wherein the first neural network is used to: extract image features of the to-be-recognized image, and use the image features to estimate a target area corresponding to the to-be-recognized image. The device according to claim 50 or 53, wherein the image features include any one of the following: a face region feature, an eye region feature, or a head pose feature. The device according to claim 54, wherein the face region feature, eye region feature or head pose feature are respectively the face region, eye region or head segmented from the to-be-recognized image Correspondingly extracted from the partial area. The device according to claim 55, wherein the eye region features include any one of the following: binocular features, left eye features or right eye features. The device according to claim 56, wherein the left-eye feature and the right-eye feature are extracted by using a Siamese network. The device according to claim 57, wherein the twin network is used for: performing dilated convolution operations on the left eye region and the right eye region respectively, so as to enhance the receptive field of the network to the image. The device according to claim 49 or 51, wherein the multiple frames of images to be identified include: multiple frames of images captured by the same shooting device at different times of the user; or, images captured by different shooting devices at different positions of the user multi-frame images. The device according to claim 46 or 53, wherein the target area is estimated by the first neural network by identifying the user's gaze direction information and/or the user's head posture information in the to-be-recognized image out. The apparatus of claim 46, wherein the structure of the first neural network is the same as the structure of the second neural model. 61. The apparatus of claim 61, wherein the parameters of the first neural network are different from the parameters of the second neural model. The apparatus according to claim 61, wherein the first neural network is a convolutional neural network and/or the second neural network is a convolutional neural network. The apparatus of claim 52, wherein the third neural network is a recurrent neural network. The apparatus of claim 46, wherein the target area is one of a plurality of set areas. The device according to claim 46, wherein the to-be-identified image comprises: an image taken of a driver in a car of a vehicle. 66. The apparatus of claim 66, wherein the target gaze direction information is used to assess whether the driver is distracted driving. The device according to claim 66, wherein the plurality of setting areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, Dashboard area or center console area. The apparatus according to claim 67, wherein the processor further implements the following steps when executing the computer program: Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output. The device according to claim 69, wherein the output mode of the prompt information includes any one of the following modes: audio, text, video, or lighting. A line-of-sight estimation device, characterized in that the device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, and the processor implements the following steps when executing the computer program: Acquiring at least one frame of an image to be identified, the image to be identified includes an image obtained by performing image collection on a user; Use a neural network to estimate the user's target line of sight direction information; wherein, the neural network is used to: after extracting at least one type of target features representing the orientation of the user's head in the to-be-recognized image, fuse the extracted target features, The target gaze direction information of the user is estimated according to the feature fusion result. The device according to claim 71, wherein the target feature comprises any one of the following: a face region feature, an eye region feature, or a head pose feature. The device according to claim 72, wherein the face region feature, eye region feature or head pose feature are respectively a face region, an eye region or a head segmented from the to-be-recognized image Correspondingly extracted from the partial area. The device according to claim 73, wherein the eye region feature comprises any one of the following: a binocular feature, a left eye feature or a right eye feature. The device according to claim 74, wherein the left-eye feature and the right-eye feature are extracted by using a Siamese network. The device according to claim 75, wherein the twin network is used for: performing a dilated convolution operation on the left eye region and the right eye region respectively, so as to enhance the receptive field of the network to the image. The device according to claim 71, wherein the target line-of-sight direction information of the user comprises: a target area, and the target area represents an area in the to-be-recognized image to which the user's head is facing. The device according to claim 71, wherein the training data of the neural network adopts an image of the target area in which the area to which the user's head is facing is used. The device according to claim 71, wherein the target gaze direction information of the user comprises: angle information of the user's target gaze direction. The device according to claim 71, wherein the to-be-identified image has multiple frames; When the processor executes the estimation of the target gaze direction information of the user by using the neural network, the following steps are implemented: Obtaining the line-of-sight direction information of the user in each frame of the to-be-recognized image; the user's line-of-sight direction information in each frame of the to-be-recognized image is estimated by the neural network corresponding to the to-be-recognized image of the frame of the to-be-recognized image; The line-of-sight direction information corresponding to each frame of the image to be recognized is fused to estimate the user's target line-of-sight direction information. The device according to claim 71, wherein the image to be recognized has multiple frames; the neural network comprises an initial sub-neural network and a fusion sub-neural network; When the processor executes the estimation of the target gaze direction information of the user by using the neural network, the following steps are implemented: For each frame of the to-be-recognized image, the initial sub-neural network is used to obtain the feature fusion result of each frame of the to-be-recognized image; the initial sub-neural network is used to: extract at least one type of target representing the user's head orientation in the to-be-recognized image After the feature, the extracted target features are fused; The fusion sub-neural network is used to fuse the feature fusion results of the images to be identified in each frame to estimate the user's target line of sight direction information. The apparatus of claim 71, wherein the neural network is a convolutional neural network. The apparatus according to claim 81, wherein the initial sub-neural network is a convolutional neural network. The apparatus according to claim 81, wherein the fusion sub-neural network is a recurrent neural network. The device according to claim 71, wherein the to-be-identified image comprises: an image taken of a driver in a vehicle. 84. The apparatus of claim 84, wherein the target gaze direction information is used to assess whether the driver is distracted driving. The device according to claim 84, wherein the plurality of setting areas include: a front windshield area, a left rearview mirror area, a right rearview mirror area, a left window area, a right window area, Dashboard area or center console area. The apparatus according to claim 71, wherein the processor further implements the following steps when executing the computer program: Using the target gaze direction information, it is determined that the user's gaze direction deviates from the set direction, and prompt information is output. The device according to claim 88, wherein the output mode of the prompt information includes any one of the following modes: audio, text, video or lighting. A vehicle, characterized in that the vehicle comprises the line-of-sight estimation device according to any one of claims 46 to 70 or the line-of-sight estimation device according to any one of claims 71 to 89 . A computer-readable storage medium, characterized in that, several computer instructions are stored on the readable storage medium, and when the computer instructions are executed, the steps of the method of any one of claims 1 to 26 are implemented. A computer-readable storage medium, characterized in that, a plurality of computer instructions are stored on the readable storage medium, and when the computer instructions are executed, the steps of the method according to any one of claims 27 to 45 are implemented.

Images