JP7633885B2 - Gaze Estimation System - Google Patents

Description

The present invention relates to a gaze estimation system.

There are two types of gaze estimation technology that use image processing: model-based methods and appearance-based methods. Gaze estimation using a model-based method involves, for example, creating a three-dimensional model that includes an eyeball model, and estimating the gaze by comparing the model with an image that includes the subject's eyes (see, for example, Patent Document 1). Gaze estimation using an appearance-based method involves estimating the gaze by comparing an image that includes the subject's eyes with multiple learning images that have been machine-learned by a learning device (see, for example, Patent Document 2).

JP 2018-88236 A JP 2019-28843 A

According to the appearance-based method, the gaze of the subject can be estimated with high accuracy by using deep learning, but the processing load on the computing device increases and a large amount of learning is required for machine learning. On the other hand, according to the model-based method, the processing load on the device is reduced compared to the appearance-based method, but it is not easy to estimate the gaze of the subject with high accuracy. Therefore, when estimating the gaze of the subject, it is desirable to achieve both a reduction in the processing load and high-accuracy gaze estimation.

The present invention has been made in consideration of the above problems, and aims to provide a gaze estimation system that can achieve both a reduction in processing load and highly accurate gaze estimation.

In order to achieve the above object, the gaze estimation system according to the present invention includes an image acquisition unit that acquires photographed images including the face of the subject in a continuous time series, and a processing unit that executes an estimation process of the gaze of the subject based on each of the acquired photographed images. The processing unit executes a first gaze estimation process that performs an appearance-based gaze estimation process based on a learning image including the face of the subject to generate first estimated gaze information, and a second gaze estimation process that performs a model-based gaze estimation process based on the learning image to generate second estimated gaze information. The processing unit has a first determination unit that determines whether the gaze of the subject is directed toward a first target range that requires relatively high accuracy or toward a second target range that is outside the first target range and does not require higher accuracy than the first target range, depending on a movement accompanied by a change in the gaze of the subject obtained from each of the photographed images. If the first determination unit determines that the gaze is directed toward the first target range, the processing unit executes the first gaze estimation process, and if the gaze is determined to be directed toward the second target range, the processing unit executes the second gaze estimation process.

The gaze estimation system according to the present invention has the advantage of being able to reduce the processing load in estimating the gaze of the subject based on each captured image while achieving highly accurate gaze estimation.

FIG. 1 is a schematic diagram illustrating an application example of a gaze estimation system according to an embodiment. FIG. 2 is a block diagram showing a schematic configuration of the line-of-sight estimation system of FIG. FIG. 3A is a state transition diagram showing an overview of the operation of the gaze estimation system of FIG. 1, and FIG. 3B is a schematic diagram showing an example of a target range to which the measurement subject directs his/her gaze. FIG. 4 is a schematic diagram showing an overview of the first gaze estimation process executed by the gaze estimation system of FIG. FIG. 5 is a flowchart showing an outline of the second gaze estimation process executed in the gaze estimation system of FIG. FIG. 6(A) is a schematic diagram showing the relationship between a three-dimensional (3D) model of the eyeball and a two-dimensional (2D) image in step S16 of FIG. 5, and FIG. 6(B) is a schematic diagram showing an overview of the iris search performed in steps S17 and S18 of FIG. 5. FIG. 7 is a flowchart showing an example of an algorithm of the gaze estimation process executed by the gaze estimation system of FIG. FIG. 8 is a schematic diagram illustrating an example of a target range to which a measurement subject directs his/her gaze in a gaze estimation system according to a modified example of the embodiment.

The following describes in detail a gaze estimation system according to an embodiment of the present invention with reference to the drawings. Note that the present invention is not limited to the embodiments described below. The components in the following embodiments include those that a person skilled in the art would easily imagine, or those that are substantially the same. Furthermore, the components in the following embodiments can be omitted, replaced, or modified in various ways without departing from the spirit of the invention.

[Embodiment]
The gaze estimation system 1 of the present embodiment shown in Fig. 1 and Fig. 2 is a system that captures an image of a subject and estimates the gaze of the subject based on the captured image including the face of the subject. In this embodiment, the gaze estimation system 1 is applied to a vehicle. The gaze estimation system 1 includes a photographing unit 12 and a processing unit 13.

The photographing unit 12 is an example of an image acquisition section (or image acquisition device) that captures the face 21 of the driver 20 of the vehicle 2 and acquires photographed images including the face 21 of the driver 20 continuously in time series. In this embodiment, the driver 20 is the subject of measurement. The photographing unit 12 includes a light source and a camera. The light source and the camera are arranged adjacent to each other on a single substrate, for example.

The light source is, for example, an LED (Light Emitting Diode) that emits near-infrared light toward the outside of the photographing unit 12. Near-infrared light is an electromagnetic wave with a wavelength of approximately 0.7 to 2.5 μm, and has a wavelength close to red visible light. For example, it is used as "invisible light" in infrared cameras and infrared communications. The light source turns on (emits infrared light) in response to a turn-on signal input from the processing unit 13, and turns off in response to a turn-off signal. The light source is placed on or near the object to be viewed.

The camera is placed near the light source and captures an image of a shooting range (e.g., an angle of view) set outside the shooting unit 12. When the driver 20 is present in the shooting range set outside the shooting unit 12, the camera captures the driver 20 illuminated by infrared light emitted from the light source. The shooting unit 12 is installed, for example, in the meter unit in front of the driver's seat 101 or on the top of the column cover in order to capture an image of the face 21 of the driver 20 seated in the driver's seat 101 in the passenger compartment 10 of the vehicle 2. The shooting unit 12 is connected to the processing unit 13 and outputs the captured image to the processing unit 13.

The camera is placed in a position where it receives the reflected light reflected in the opposite direction to the direction of emission of the infrared light emitted from the light source. When the camera is placed in a position where it receives the reflected light reflected in the opposite direction to the direction of emission of the infrared light, the optical axis of the camera and the optical axis of the light source overlap. The optical axis of the camera and the optical axis of the light source overlap means that both optical axes are coaxial, but it is sufficient that both optical axes are parallel and the reflected light can be received. It is preferable that the camera can capture at least the entire face 21 with sufficient resolution. The capture range can be determined based on the range to be detected that is required by the application. Examples of applications include measuring the interest of vending machines, automatically counting the number of viewers of advertisements such as digital signage, and turning on (or increasing the brightness of) a display system only when it is viewed.

The processing unit 13 is an example of a processing unit (or processing device) that performs an estimation process of the line of sight of the driver 20 based on the captured image acquired by the photographing unit 12. The processing unit 13 has, for example, a processing circuit (not shown) that realizes various processing functions in the gaze estimation system 1. The processing circuit is realized, for example, by a processor. The processor means, for example, a circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). The processing unit 13 realizes each processing function, for example, by executing a program read from a memory circuit (memory unit) not shown.

As shown in FIG. 2, the processing unit 13 includes a first gaze estimation processing unit 14A, a second gaze estimation processing unit 14B, a first judgment unit 15A, a second judgment unit 15B, and a third judgment unit 15C.

The first gaze estimation processing unit 14A of the processing unit 13 executes a first gaze estimation process that generates first estimated gaze information by performing a gaze estimation process by an appearance-based method based on a learning image including the face 21 of the driver 20. The first estimated gaze information is, for example, the gaze angle of the gaze 24 of the driver 20. The first gaze estimation process is executed by the first gaze estimation processing unit 14A. The gaze estimation technology by the appearance-based method is described in, for example, the following document.
"High-precision gaze estimation using appearance-based methods for specific environment applications, Qu Manting (Keio University), Takahashi Yuki (Yazaki Corporation), et al., 26th Symposium on Image Sensing (SSII2020), IS2-18, June 2020."

Gaze estimation using appearance-based methods uses the subject's eye image itself as input information, learns the combination of gaze and eye image through machine learning, and estimates the gaze position for a new eye image. Machine learning methods include, for example, Convolutional Neural Networks (CNN), a deep learning method. CNN is a type of DNN (Deep Neural Network), which is a multi-layered pattern recognition method called a neural network, that is compatible with two-dimensional data and has been reported to have high pattern recognition capabilities for images. To perform learning with CNN, a correct label corresponding to the input image is required. In this case, a face image is given as input information and the gaze angle is given as the correct label.

In estimating gaze using an appearance-based method, it is necessary to create a learning dataset and generate a model by machine learning. At this time, the accuracy of the learning dataset affects the accuracy of the learning. Therefore, when creating the learning dataset, it is necessary to ensure that the subject (or test subject) is looking accurately at the visual target. For example, as shown in FIG. 4, the subject wears a wearable gaze measurement device and gaze measurement results are collected. Image 31 was obtained by photographing the subject while wearing the gaze measurement device. On the other hand, when actually estimating the gaze of the subject, a face image without the gaze measurement device is input, so in order to make it closer to the image during actual use, a process is performed to paint the arm part of the gaze measurement device that is over the face image with the same color as the face, as shown in image 32.

The second gaze estimation processing unit 14B of the processing unit 13 executes a second gaze estimation process that performs gaze estimation processing using a model-based method based on a learning image including the face 21 of the driver 20 to generate second estimated gaze information. The second estimated gaze information is, for example, the gaze angle of the gaze 24 of the driver 20. The second gaze estimation process is executed by the second gaze estimation processing unit 14B. A gaze estimation technique using a model-based method is disclosed, for example, in JP 2018-88236 A.

The process shown in FIG. 5 is an example of the second gaze estimation process, and each step is performed sequentially by the processing unit 13 executing a program read from the storage unit.

In step S12, the photographing unit 12 photographs an image including the face of the driver 20 and outputs an image signal. The processing unit 13 captures one frame of the image signal from the photographing unit 12.

In step S13, the processing unit 13 converts the data format of the image (pre-processing), including grayscaling. For example, for each pixel position in one frame, 8-bit data expressing brightness in a gradation range of "0 to 255" is generated as a two-dimensional (2D) array of image data arranged vertically and horizontally in accordance with the scanning direction within the frame at the time of shooting.

In step S14, the processing unit 13 performs face detection using, for example, the "Viola-Jones method" based on the image converted in step S13, and extracts a face image including a face from one frame of two-dimensional image data. That is, the face image is extracted using a detector that is characterized by the shading difference of the face and that is created by learning using "Boosting". The technology of the "Viola-Jones method" is described in, for example, the following document.
“Viola, Paul and Michael J. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features”, Proceedings of The 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. Volume: 1, pp. 511-518.

In step S15, the processing unit 13 detects the eye area from the face image extracted in step S14, for example, using the Viola-Jones method described above.

In step S16, when the processing unit 13 detects the gaze using a model-based method that uses a 3D eyeball model described below, it estimates the eyeball center. Here, the central coordinates of the rectangular area of the eye detected in step S15 are assumed to be the eyeball center. For example, it is also possible to use a method of determining the eyeball center based on the positions of the outer corner and inner corner of the eye, or a method of estimating the skeleton from facial feature points and simultaneously calculating the eyeball center position.

In step S17, the processing unit 13 performs a rough search for the iris (pupil or iris) by applying a template matching technique to the data of the rectangular region of the eye detected in step S15. Specifically, a black circle image is matched as a template to a binarized image of the eye image in which the area around the eye has been cut out, and the coordinates of the image center (center of the black circle) of the black circle image with the highest likelihood are set as the center position of the iris in the eye image, and the radius of the black circle image with the highest likelihood is set as the radius of the iris in the eye image. The processing of step S17 is performed to roughly estimate the center position and radius of the iris in the eye image.

In step S18, the processing unit 13 uses the center position and radius of the iris found in step S17 and applies a particle filter technique to detect the center position and radius of the iris with higher accuracy.

In step S19, the processing unit 13 obtains numerical data on the eyeball center coordinates and the coordinates of the center position of the iris for one frame of image through the processes in steps S13 to S18 described above, and outputs this numerical data. The direction of gaze can be identified from the eyeball center coordinates and the coordinates of the center position of the iris. Furthermore, by repeating the loop processes in steps S11 to S20 until the video output by the photographing unit 12 is interrupted, gaze detection can be achieved in real time. When using this data to calculate the eyeball rotation angle from the coordinates on the image plane, a conversion is performed using a 3D eyeball model as shown in Figure 6 (A).

In the 3D eyeball model shown in FIG. 6A, the eyeball E is a sphere represented by the eyeball center O and the eyeball radius R. In addition, on the surface of the eyeball E, there is a circular iris F that constitutes the black eye, and in the center of the iris F, there is a circular pupil G. The direction of the line of sight can be specified as the direction from the eyeball center O toward the center of the iris F or the pupil G, and can be represented by the rotation angle yaw with respect to a reference direction in the horizontal plane, and the rotation angle pitch with respect to a reference direction in the up-down direction. In addition, when the eyeball center O is used as a reference, the central coordinates of the iris F or the pupil G can be represented by the eyeball radius R, yaw, and pitch. On the other hand, since the image captured by the photographing unit 12 represents a two-dimensional plane, when applying a two-dimensional image captured by the photographing unit 12 to the 3D eyeball model, it is necessary to perform two-dimensional/three-dimensional mutual conversion. For example, the conversion is performed using the following formula.

X=-R×cos(pitch)×sin(yaw)...(1)
Y=R×sin(pitch)...(2)
X: distance in the x direction from the eyeball center O on the two-dimensional image plane Y: distance in the y direction from the eyeball center O on the two-dimensional image plane

Next, the process of searching for the iris from a face image will be described with reference to FIG. 6B. In step S41, the processing unit 13 uses the result of step S15 in FIG. 5 to cut out the eyes and a rectangular area around them from the two-dimensional image data of one entire frame, and obtains data D41.

In step S42, the processing unit 13 generates data D42 by binarizing the data D41 acquired in step S41 so that the gradation for each pixel is only two values, black/white, and then performs template matching on this data D42 in step S17 in FIG. 5. That is, an image of a black circle shape similar to the shape of the iris is used as a template, and while scanning the image of data D42 with this template, the approximate position of the iris with the most similar features is searched for, and the position and the radius or diameter of the iris are identified.

In step S43, the processing unit 13 applies Sobel filter processing to the eye data D41 acquired in step S41. Specifically, the eye data D41 is scanned horizontally from left to right, and edges are detected by outputting black (tone value: 0) in areas where there is no change in luminance, and approaching white (tone value: 255) as the gradient of the luminance change increases. This results in the acquisition of eye image data D43 from which edges have been extracted.

In step S44, the processing unit 13 performs particle filter processing on the eye image data D43 obtained in step S43, starting from the approximate position of the iris obtained in step S42. Iris detection using the particle filter is performed in the following order [1] to [4].

[1] The processing unit 13 searches for a rough position of the iris by template matching. [2] The processing unit 13 calculates the eyeball rotation angle from the iris position in [1] above based on a 3D eyeball model.
[3] The processing unit 13 scatters particles for sampling iris ellipse candidates in the vicinity of the rough eyeball rotation angle obtained in [2] above.
[4] The processing unit 13 calculates the likelihood at the iris ellipse candidate positions scattered in [3]. Here, the likelihood is calculated by summing the scores of all points on the circumference of the iris ellipse candidate 44a, calculating +1 for points extracted as edges by referring to the edge image D44 of the Sobel filter, and +0 for points not extracted as edges.
[5] Calculate the likelihood for the scattered particles, and output the detection result after capturing the trend of change in the likelihood with respect to the scattering position. In other words, from the scattering results of multiple particles, the point at which the likelihood is maximized with respect to the scattering position change is output as the detection result.

For the vicinity of the eyeball rotation angle in [3] above, multiple patterns are created by adding blur due to random numbers to the yaw and pitch values of the 3D eyeball model shown in Figure 6 (A). At this time, the amount of change in blur detected in the next frame is generated from the detection result of the previous frame. In other words, the greater the difference in the detection result from the median of the random numbers, the greater the amount of change in blur due to the random numbers.

As shown in FIG. 3A, the processing unit 13 executes model-based gaze estimation (second gaze estimation process), and executes appearance-based gaze estimation (first gaze estimation process) if the result of the gaze estimation process satisfies condition A. On the other hand, when the processing unit 13 executes the first gaze estimation process, if the result of the face direction calculation process executed before executing the first gaze estimation process satisfies condition B, the processing unit 13 executes the second gaze estimation process. Condition A is, for example, a gaze angle estimated in the second gaze estimation process within ±θ°. θ° (yaw, pitch) is a threshold value for transitioning to appearance-based gaze estimation based on the gaze angle obtained in the model-based gaze estimation, and indicates a corresponding range in the appearance-based gaze estimation. Condition B is a face direction angle calculated in the face direction calculation process is ±φ° or more. φ° (yaw, pitch) is a threshold value for the face direction angle for transitioning from appearance-based gaze estimation to model-based gaze estimation. φ° can be set based on the corresponding range θ° in appearance-based gaze estimation. For example, it can be set based on a sharing ratio that has been considered regarding the relationship between the face direction when looking at an object and the gaze angle. Details of the sharing ratio are shown in, for example, the following document. Note that the processing unit 13 performs initial setting immediately after the system is started, and executes the second gaze estimation process after the initial setting.
"Analysis of the Coordination of Head and Eye Movements When Gazing at a Two-Dimensional Indicator, Mitsuho Yamada, IEICE Transactions on Computer Science and Technology Vol. J75-D2 No. 5 pp. 971-981 May 1992"

The first determination unit 15A determines whether the line of sight 24 of the driver 20 is directed toward the first target range 41 or the second target range 42, depending on the movement accompanied by a change in the line of sight 24 of the driver 20 obtained from each captured image. The movement accompanied by a change in the line of sight 24 of the driver 20 obtained from each captured image is, for example, a movement of turning the face or a movement of turning the body. In this embodiment, the movement is a movement of turning the face. The first determination unit 15A calculates the facial direction angle of the driver 20 from each captured image, and determines whether the line of sight 24 is directed toward the first target range 41 or the second target range 42, based on the facial direction angle.

The first target range 41 is set to a range within the visible range of the driver 20 that requires a relatively high-precision gaze estimation. The first target range 41 is, for example, the driving visual field range, as shown in FIG. 3B. In the driving visual field range, it is required to estimate the gaze direction, for example, whether the driver 20 saw the brake lights of the vehicle ahead, whether the driver saw a red light, or whether the driver saw a pedestrian at a crosswalk in the distance. The second target range 42 is a range within the visible range of the driver 20 that is outside the first target range 41 and does not require a gaze estimation with a higher degree of precision than the first target range 41. The second target range 42 is, for example, a range into which the driver 20 looks when checking for safety or looking aside.

When the second judgment unit 15B obtains the second estimated gaze information, it judges whether the gaze 24 is directed toward the first target range 41 or the second target range 42 based on the second estimated gaze information.

The third determination unit 15C determines whether the gaze estimation process executed immediately before was the first gaze estimation process or the second gaze estimation process.

Next, the algorithm of the gaze estimation process executed by the gaze estimation system 1 will be described with reference to Figs. 3 and 7. In the process shown in Fig. 7, each step is performed sequentially by the processing unit 13 executing a program read from the memory unit in response to, for example, power-on (e.g., turning on the IGN of the vehicle 2).

In step S31, the photographing unit 12 photographs an image including the face of the driver 20 and outputs an image signal. The processing unit 13 captures one frame of the image signal from the photographing unit 12. At this time, the processing unit 13 converts the data format of the image, including grayscaling, and changes the size, as necessary.

In step S32, the processing unit 13 performs face detection using, for example, the above-mentioned "Viola-Jones method" based on the captured image captured in step S31, and extracts a facial image including a face from one frame of two-dimensional image data.

In step S33, the processing unit 13 performs eye detection in the face image extracted in step S32. The processing unit 13 can detect the position of the eyes using, for example, the above-mentioned "Viola-Jones method", but in order to calculate the face direction in a later step, it is preferable to perform a method of detecting facial feature points and identifying a rectangular part that includes the eye feature points. For example, a general facial feature point detection algorithm shown on the following site may be used. Facial point annotations: https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/

In step S34, the processing unit 13 calculates the face direction in the face image extracted in step S32. The algorithm executed in step S33 includes an algorithm for calculating the face direction. This is a method of estimating the rotation angle (yaw, pitch, roll) of the face direction by creating a standard face model in advance and comparing the position of each point corresponding to the detected two-dimensional feature point arrangement. Note that the face direction angle is used in a later step, so it is temporarily stored in a memory unit in the processing unit 13, for example. Note that the algorithm executed in steps S32 to S34 may be realized using software other than the above-mentioned method.

In step S35, the processing unit 13 determines whether or not there is a gaze estimation process that has been executed immediately before. The processing unit 13 determines, for example, whether or not a software usage log is recorded in the storage unit. Here, the software usage log includes a log corresponding to the first gaze estimation process and a log corresponding to the second gaze estimation process. If the software usage log is not recorded in the storage unit, the processing unit 13 determines that there is no gaze estimation process that has been executed immediately before, and proceeds to step S41. The case where the software usage log is not recorded in the storage unit includes, for example, a case where the gaze estimation process is executed for the initial frame or a case where the system is initialized. Alternatively, the face 21 or eyes 22 cannot be detected in the previous face detection or eye detection, that is, the face 21 of the driver 20 is facing to the side or backward, and the face 21 or eyes 22 are not present within the image range. If the software usage log is recorded in the storage unit as a result of the determination in step S35, the processing proceeds to step S36.

In step S36, the processing unit 13 determines whether the gaze estimation process executed immediately before was the first gaze estimation process or the second gaze estimation process. Step S36 is executed by the third determination unit 15C. The processing unit 13 refers to the software usage log recorded in the memory unit and determines whether it corresponds to the first gaze estimation process or the second gaze estimation process. If the software usage log corresponds to the first gaze estimation process, the processing unit 13 proceeds to step S37. On the other hand, if the software usage log corresponds to the second gaze estimation process, the processing unit 13 proceeds to step S41.

In step S37, the processing unit 13 determines whether the face direction angle is ±φ° or more. Step S37 is executed by the first determination unit 15A. The face direction angle is the one calculated in step S34. If the processing unit 13 determines that the face direction angle is ±φ° or more, it determines that the line of sight 24 of the driver 20 is directed toward the second target range 42, and proceeds to step S41. On the other hand, if the processing unit 13 determines that the face direction angle is not ±φ° or more, it determines that the line of sight 24 of the driver 20 is directed toward the first target range 41, and proceeds to step S38.

In step S38, the processing unit 13 executes the first gaze estimation process and proceeds to step S39.

In step S39, the processing unit 13 outputs the result of the gaze estimation process that was performed and proceeds to step S40.

In step S40, the processing unit 13 records a software usage log indicating the gaze estimation process that was performed in the memory unit and returns to step S31.

In step S41, the processing unit 13 executes the second gaze estimation process and proceeds to step S42. The processing unit 13 temporarily records the result of the executed gaze estimation process in the memory unit.

In step S42, the processing unit 13 determines whether the gaze angle is within ±θ°. Step S42 is executed by the second determination unit 15B. The gaze angle is the one estimated in step S41. If the processing unit 13 determines that the gaze angle is within ±θ°, it determines that the gaze 24 of the driver 20 is directed toward the first target range 41, erases the result of the gaze estimation process temporarily recorded in step S41, and proceeds to step S38. On the other hand, if it determines that the gaze angle is not within ±θ°, it determines that the gaze 24 of the driver 20 is directed toward the second target range 42, and proceeds to step S39.

The gaze estimation system 1 described above determines whether the gaze 24 of the driver 20 obtained from the captured image is directed toward a first target range 41, which requires a relatively high level of accuracy, or toward a second target range 42, which is outside the first target range 41 and does not require a higher level of accuracy than the first target range 41. As a result, if it is determined that the gaze 24 is directed toward the first target range 41, a first gaze estimation process is executed in which a gaze estimation process using an appearance-based method is performed to generate first estimated gaze information. On the other hand, if it is determined that the gaze 24 is directed toward the second target range 42, a second gaze estimation process is executed in which a gaze estimation process using a model-based method is performed to generate second estimated gaze information.

With the above configuration, when the driver 20 is directing his/her gaze to the first target range 41, which requires high accuracy, the gaze can be estimated with high accuracy using the appearance-based method, and when the driver is directing his/her gaze to the second target range 42, which does not require high accuracy, the gaze can be estimated using the model-based method, which has a lower processing load than the appearance-based method. As a result, the processing load can be reduced compared to when estimating the gaze using only the appearance-based method, and the gaze can be estimated with high accuracy compared to when estimating the gaze using only the model-based method, so that both the reduction in processing load and high accuracy gaze estimation can be achieved. In addition, when the gaze estimation system 1 is applied to a vehicle, it is possible to estimate the gaze in the driving field of view in front of the vehicle and the gaze outside the driving field of view, such as the driver 20 checking for safety or looking aside.

Furthermore, when the action is a facial movement of the driver 20, the gaze estimation system 1 calculates the facial direction angle of the driver 20 from each captured image, and determines whether the gaze 24 is directed toward the first target range 41 or the second target range 42 based on the facial direction angle. As a result, the processing load can be reduced by determining which target range the gaze 24 is directed toward based on the facial direction angle of the driver 20, compared to, for example, performing the second gaze estimation process for each image and determining whether the gaze 24 of the driver 20 is directed toward the first target range 41 or the second target range 42.

Furthermore, the gaze estimation system 1 determines whether the gaze 24 is directed toward the first target range 41 or the second target range 42 based on the second gaze estimation gaze information, and executes the first gaze estimation process if it determines that the gaze 24 is directed toward the first target range 41. As a result, even if it erroneously determines that the gaze 24 of the driver 20 is directed toward the second target range 42 instead of the first target range 41 and performs gaze estimation in the second gaze estimation process, it is possible to execute the first gaze estimation process and estimate the gaze directed toward the first target range 41 with high accuracy.

The gaze estimation system 1 also determines whether the gaze estimation process executed immediately before was the first gaze estimation process or the second gaze estimation process, and if the gaze estimation process executed immediately before was the second gaze estimation process, executes the second gaze estimation process, and if it was the first gaze estimation process, makes a determination by the first determination unit 15A. As a result, for example, if the gaze estimation process executed immediately before was the second gaze estimation process, the second gaze estimation process is performed without making a determination by the first determination unit 15A, thereby reducing the amount of processing for determination by the first determination unit 15A.

[Modification]
In the above embodiment, the gaze estimation system 1 is applied to a vehicle 2 such as an automobile, but is not limited thereto, and may be applied to, for example, a ship or an aircraft other than the vehicle 2. In addition, the gaze estimation system 1 is separated into the photographing unit 12 and the processing unit 13, but is not limited thereto, and may be configured as an integrated unit.

The gaze estimation system 1 may also be applied to digital signage. For example, by installing the photographing unit 12 on the digital signage, it is possible to analyze where passersby looked on the digital signage. In the example shown in FIG. 8, the photographing unit 12 is installed on the top of the central digital signage 51. The photographing unit 12 may be installed at the bottom, but it is necessary to create a gaze database that corresponds gaze angle information and face images to learn a trained model used for gaze estimation by the appearance-based method according to the installation position and the range in which the measurement subject appears. In such a gaze estimation system, within the digital signage 51 in which the photographing unit 12 is installed, it is possible to accurately estimate which part of the digital signage 51 is focused on by using gaze estimation by the appearance-based method. Furthermore, within the digital signage outside 52, it is possible to estimate gaze information such as which of the digital signage outside 52 adjacent to the left and right of the digital signage 51 is being looked at by using gaze estimation by the model-based method. In addition, by recording the gaze position and the duration of gaze, it is possible to measure the degree of attention to the displayed content.

In the above embodiment, the processing circuit has been described as having each processing function realized by a single processor, but this is not limited to the above. The processing circuit may be configured by combining multiple independent processors, each of which executes a program to realize each processing function. The processing functions of the processing circuit may be realized by distributing or integrating them as appropriate in a single or multiple processing circuits. The processing functions of the processing circuit may be realized in whole or in any part by a program, or may be realized as hardware using wired logic or the like.

The program executed by the processor described above is provided by being pre-installed in a storage circuit or the like. This program may also be provided by being recorded on a computer-readable storage medium in a format that can be installed on these devices or in a format that can be executed. This program may also be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network.

REFERENCE SIGNS LIST 1 Gaze estimation system 2 Vehicle 12 Shooting unit 13 Processing unit 14A First gaze estimation processing unit 14B Second gaze estimation processing unit 15A First determination unit 15B Second determination unit 15C Third determination unit 20 Driver 21 Face 22 Eyes

Claims (4)

an image acquisition unit that acquires photographed images including a face of a subject in a time series;
a processing unit that executes an estimation process of the line of sight of the measurement subject based on each of the acquired photographic images,
The processing unit includes:
a first gaze estimation process for performing a gaze estimation process by an appearance-based method based on a learning image including a face of the measurement subject to generate first estimated gaze information, and a second gaze estimation process for performing a gaze estimation process by a model-based method based on the learning image to generate second estimated gaze information,
a first determination unit that determines, according to a movement accompanied by a change in the line of sight of the measurement subject obtained from each of the captured images, whether the line of sight is directed toward a first target range requiring a relatively high degree of accuracy or toward a second target range outside the first target range and not requiring a higher degree of accuracy than the first target range;
When the first determination unit determines that the line of sight is directed toward the first target range, the first line of sight estimation process is executed;
When it is determined that the line of sight is directed toward the second target range, the second line of sight estimation process is executed.
The gaze estimation system is characterized by: The movement is a facial movement of the measurement subject,
The first determination unit is
calculating a face direction angle of the measurement subject from each of the captured images, and determining whether the line of sight is directed toward the first target range or the second target range based on the face direction angle;
The gaze estimation system according to claim 1 . The processing unit includes:
a second determination unit that, when the second estimated gaze information is obtained, determines whether the gaze is directed toward the first target range or the second target range based on the second estimated gaze information;
When the second determination unit determines that the line of sight is directed toward the first target range, the first line of sight estimation process is executed.
The gaze estimation system according to claim 1 . The processing unit includes:
a third determination unit that determines whether a previously executed gaze estimation process is the first gaze estimation process or the second gaze estimation process,
When the third determination unit determines that the gaze estimation process executed immediately before is the second gaze estimation process, the second gaze estimation process is executed, and when the third determination unit determines that the gaze estimation process executed immediately before is the first gaze estimation process, a determination is made by the first determination unit.
The gaze estimation system according to claim 3 .

Images