JP5161685B2 - Gaze measurement apparatus and program - Google Patents

Description

  The present invention relates to a line-of-sight measurement apparatus and a program, and more particularly, to a line-of-sight measurement apparatus and a program that perform automatic calibration.

  2. Description of the Related Art Conventionally, there is known an apparatus that issues an alarm for a side-view driving or the like by mounting a line-of-sight measurement device on a vehicle and measuring the line of sight of a driver. In such a line-of-sight measurement device, it is necessary to calibrate so that the actual line-of-sight direction and the measured line-of-sight direction coincide with each other. A correction coefficient is calculated from the relationship between the gaze direction and the actual gaze position.

However, such a calibration process places a heavy burden on the driver. Therefore, for example, in the gaze direction measuring device described in Patent Document 1, an operation signal is generated when an operation of a device such as a winker lever or a shift lever is performed, and the calculation is performed when the operation signal is generated. It has been proposed to perform automatic calibration with a correction coefficient calculated using line-of-sight data. This is to take advantage of an increase in gaze at a specific position (for example, gaze on the right side mirror when operating the right winker) when operating a specific device.
Japanese Patent No. 3785669

  However, for example, when the right winker is turned on, the right side mirror is not always watched. In the gaze direction measuring device of Patent Document 1, the gaze data is accumulated after the device is operated. Therefore, there is a problem that there is a possibility that a lot of line-of-sight data that is not appropriate as line-of-sight data for calculating the correction coefficient may be included.

  The present invention is made in order to solve the above problem, and even when automatic calibration is performed, by calculating a correction coefficient using line-of-sight data with high reliability of gazing at a specific position, An object of the present invention is to provide an eye gaze measuring apparatus and program capable of improving the accuracy of eye gaze measurement.

  In order to achieve the above object, the eye gaze measurement apparatus of the present invention obtains eye gaze data indicating the eye gaze direction of the operator based on image data of an eye area obtained by photographing the operator who operates the device. Line-of-sight data calculation means to calculate, acquisition means for acquiring operation data when the operator operates the device, line-of-sight data calculated by the line-of-sight data calculation means, and operation data acquired by the acquisition means And a line of sight data from a predetermined time before the start time of the specific operation to the device from which the operation data stored in the storage means is acquired to the end time, An extracting unit that extracts the line-of-sight data stored in the storage unit, and the gaze position of the operator obtained based on the distribution of the line-of-sight data extracted by the extracting unit performs the specific operation To match the predetermined fixation position as a position to watch, is configured to include a, a correction coefficient calculating means for calculating a correction coefficient for correcting the gaze data calculated by said calculation means.

  Further, the eye gaze measurement program of the present invention calculates the eye gaze data indicating the eye gaze direction of the operator based on the image data of the eye area obtained by photographing the operator who operates the device. Data calculation means, acquisition means for acquiring operation data when the operator operates the device, line-of-sight data calculated by the line-of-sight data calculation means, and operation data acquired by the acquisition means Storage means for storing in association with a series; line-of-sight data from a predetermined time before the start time of a specific operation to the device from which the operation data stored in the storage means is acquired to an end time is stored in the storage means. The extraction means for extracting from the stored line-of-sight data, and the gaze position of the operator obtained based on the distribution of the line-of-sight data extracted by the extraction means performs the specific operation. To match the predetermined fixation position as a position to watch the a program for functioning as a correction coefficient calculating means for calculating a correction coefficient for correcting the gaze data calculated by said calculation means.

  According to the line-of-sight measurement apparatus and program of the present invention, the line-of-sight data indicating the line-of-sight direction of the operator based on the image data of the eye area obtained by photographing the operator who operates the device Gaze measurement is performed by calculating. In order to calibrate the line-of-sight measurement, the acquisition unit acquires operation data when the operator operates the device, the line-of-sight data calculated by the line-of-sight data calculation unit, and the operation data acquired by the acquisition unit Are associated with the time series and stored in the storage means.

  At the time of calibration, the extraction means uses the line-of-sight data stored in the storage means for the line-of-sight data from a predetermined time before the start time of the specific operation to the device from which the operation data stored in the storage means was acquired. The gaze position of the operator obtained based on the distribution of the line-of-sight data extracted by the extraction means and the correction coefficient calculation means coincides with the gaze position predetermined as the position to gaze at when performing a specific operation In this manner, a correction coefficient for correcting the line-of-sight data calculated by the calculating means is calculated.

  In this way, by extracting the line-of-sight data from a predetermined time before the start point of the specific operation to the end point, appropriate line-of-sight data associated with the operation of the device such as driving behavior is extracted. Even in the case of automatic calibration, the correction coefficient can be calculated using line-of-sight data with high reliability that a specific position is being watched, and the accuracy of line-of-sight measurement can be improved.

  In addition, the line-of-sight measurement device and the program of the present invention may be the line-of-sight data from the predetermined time before the start time of the specific operation to the end time of the device for which the operation data stored in the storage means has been acquired. In addition, line-of-sight data distributed in a predetermined area can be extracted from line-of-sight data stored in the storage means.

  When the operator gazes at a specific position, the line-of-sight data stops, and the distribution of the line-of-sight data within a predetermined region is concentrated. Therefore, the line-of-sight data from the predetermined time before the start time of the specific operation to the end of the specific operation for the device from which the operation data stored in the storage means is acquired to the storage means By extracting from the stored line-of-sight data, unstable data included in the line-of-sight data is removed, and the correction coefficient can be calculated more accurately.

  As described above, according to the present invention, even when automatic calibration is performed, the accuracy of line-of-sight measurement is improved by calculating a correction coefficient using line-of-sight data with high reliability that a specific position is being watched. The effect that it can be made is acquired.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, a case where the line-of-sight measurement device of the present invention is mounted on a vehicle and the line of sight of the driver is measured will be described.

  As shown in FIG. 1, the line-of-sight measurement device 10 according to the present embodiment includes a camera 12 that captures a face image including a driver's eye area, an operation sensor 14 that detects an operation of a device including a driver's driving behavior, and The computer 16 which performs control which calibrates gaze measurement based on operation of the driver | operator's apparatus is provided.

  The camera 12 captures a driver's face image and generates an image signal, an A / D conversion unit that converts an image signal that is an analog signal generated by the imaging unit into a digital signal, and A / D conversion An image memory for temporarily storing the received image signal.

  The operation sensor 14 detects the operation of the device including the driving behavior of the driver. For example, the winker sensor that detects the winker operation, the steering angle sensor that detects the steering amount of the steering, and the brake that detects the depression amount of the brake pedal. Various sensors for detecting the operation of the device accompanying the driving action, such as a sensor, an accelerator sensor that detects the amount of depression of the accelerator pedal, a speed sensor that detects the speed of the vehicle, and a shift lever sensor that detects the setting of the shift lever, Various sensors that detect that devices such as a car audio, a car air conditioner, and a car navigation system are operated are included.

  The computer 16 stores a CPU 24 that controls the entire line-of-sight measurement device 10, a ROM 26 that stores various programs such as a correction coefficient calculation processing program that will be described later, a RAM 28 that temporarily stores data as a work area, and various types of information. It includes a memory 30 as storage means, an I / O (input / output) port 32, and a bus connecting them. The camera 12 and the operation sensor 14 are connected to the I / O port 32.

  Next, the processing routine of the correction coefficient calculation program in the present embodiment will be described with reference to FIG.

  In step 100, an image taken at time T by the camera 12 is captured, and in step 102, line-of-sight data is calculated based on the image data of the captured image. Here, the line-of-sight data is data indicating the line-of-sight direction represented by the horizontal line-of-sight angle φ and the vertical line-of-sight angle θ with reference to the line-of-sight angle when the driver is gazing substantially in front. The gaze direction is measured based on the image feature extracted from the image data of the eye area of the captured image. In addition, the driver's face image is captured by one or more cameras, facial features such as eyes and nose are detected by image processing, and the line-of-sight direction is measured from the positional information of the feature portion, the deformed position, etc. Also good.

  Next, in step 104, the winker state at the time T detected by the operation sensor 14, the steering angle, the brake pedal depression amount, the accelerator pedal depression amount, the vehicle speed, the shift lever state, car audio, car stereo, car Captures operation data indicating the operation state such as navigation.

  Next, in step 106, the line-of-sight data at time T calculated in step 102 and the operation data at time T captured in step 104 are associated with each other and stored in a predetermined area of the memory 30.

  Next, in step 108, it is determined whether or not it is time to calculate the correction coefficient and calibrate the gaze measurement. This determination can be made based on whether or not a predetermined time has elapsed since the engine was started, or can be determined based on whether or not a predetermined time has elapsed since the last calibration. Further, calibration is performed when specific operation data, for example, operation data indicating that the right winker has changed from OFF to ON is more than a predetermined number of operation data stored in a predetermined area of the memory 30. You may make it judge that it is timing. If it is time to perform calibration, the process proceeds to step 110. If it is not time to perform calibration, the process returns to step 100, and accumulation of line-of-sight data and operation data is repeated.

  In step 110, line-of-sight data corresponding to operation data when a specific operation is performed is extracted. The specific operation may be any operation that involves gazing at a specific position while the driver performs a series of operations for the specific operation, for example, the left lane with gazing at the rearview mirror or the right side mirror. Changing the lane from to the right lane can be a specific operation.

  When changing the lane from the left lane to the right lane, the driving action is taken such that the right turn signal is turned on, the steering is rotated to the right, and then the steering is rotated to the left to return to the straight traveling state. Therefore, the operation data stored in the memory 30 is searched for the operation data corresponding to the driving action based on the operation data indicating the state of the right winker and the steering angle, and in the series of the searched operation data. The time when the right winker is turned on (T = t1) and the time when the steering angle becomes a value indicating right rotation and the steering angle becomes substantially 0 (straight forward state) (T = t2) are confirmed, and (t1 The line-of-sight data from -Δt) to t2 is extracted.

  Here, the point of time when the line-of-sight data extraction is started is (t1−Δt), since there are many cases in which gaze at a specific position related to the specific operation increases before performing a specific operation. The line-of-sight data can be extracted. For example, in the case of the above lane change, it is considered that there is often a driving action of turning on the right winker after confirming the rearview mirror and the side mirror, so at the time t1 when the right winker is turned on. The line-of-sight data from before the predetermined time Δt is extracted. As a result, more line-of-sight data can be extracted when the rear-view mirror or side mirror at the specific position is being watched.

  FIG. 3 shows an example of a plot in which the line-of-sight data is plotted with the horizontal axis of sight angle φ in the horizontal axis and the vertical axis of sight angle θ in the vertical axis. FIG. 6A is a plot of all the line-of-sight data accumulated at the time of step 108, and FIG. 4B shows a specific operation (in this case, from the left lane to the right by the processing of step 110). The line-of-sight data extracted corresponding to the operation data when the lane change to the lane is performed is plotted. As described above, when all the line-of-sight data is plotted, the line-of-sight data is largely distributed over a wide range. However, when only the line-of-sight data corresponding to specific operation data is plotted, the line-of-sight data is locally distributed to some extent.

  In FIG. 6B, the line-of-sight data is distributed roughly in three areas. The line-of-sight data distributed in a predetermined area in the vicinity of φ = 0 and θ = 0 is obtained by paying close attention to the front. It is thought that it is the line-of-sight data at the time of being. When changing lanes, taking into account the frequent operation of confirming the rear with the rearview mirror and side mirror, the line-of-sight data gazing at the rearview mirror in a predetermined area in the upper left from the positional relationship with the front It is considered that a large amount of eye gaze data is distributed, and it is considered that a large amount of line-of-sight data gazing at the side mirror is distributed in a predetermined area at the lower right.

  Next, in step 112, the line-of-sight data that is stopped is extracted. The measured line-of-sight data includes line-of-sight data when the line of sight is moving between the front and a specific gaze position. In addition, the distribution of line-of-sight data varies depending on the accuracy of the measuring device and the behavior of the driver. Therefore, the line-of-sight data distributed in a predetermined region is extracted by extracting only the line-of-sight data in which the line of sight is stopped at the front and a specific gaze position.

  Specifically, for M line-of-sight data in a predetermined section (T = t (n) to T = t (n + M)), Euclidean between the two line-of-sight data in the section is obtained by the expression (1). The distance is obtained for all the combinations of the line-of-sight data, and when the maximum value is equal to or less than the predetermined value, the line-of-sight data in the section is set as the stopped line-of-sight data.

However, x _i is a line-of-sight data vector (φ _i , θ _i ) when T = _i , x _j is a line-of-sight data vector (φ _j , θ _j ) when T = j, and C is a predetermined value Value.

  FIG. 3C shows a state where only the line-of-sight data that is stationary from the line-of-sight data plotted in FIG. 3B is plotted. Compared to FIG. 5B, the distribution of the line-of-sight data to the predetermined area is clearer.

  Next, in step 114, the center position of the distribution of the line-of-sight data distributed in the predetermined area with the line-of-sight data corresponding to the specific operation data is calculated as the specific gaze position. In the case of the above lane change, the positions of the rearview mirror and the right side mirror are specific gaze positions. Assuming that the distribution of the line-of-sight data shown in FIG. 3C is formed by a mixture of three normal distributions including the front, each average vector of the three normal distributions is obtained using the EM algorithm. Are the positions of the front, rearview mirror, and right side mirror that are gaze positions. Note that the gaze position calculation is not limited to the case where the EM algorithm is used, and the k-means method or the like may be used.

  Next, in step 116, a correction coefficient is calculated so that the gaze position calculated in step 114 matches a predetermined known gaze position.

  A shift due to parallel movement, enlargement / reduction, and rotation occurs between the calculated line-of-sight data and the reference line-of-sight data. The parallel movement is a shift caused by the parallel movement of the line-of-sight data when the head position moves left and right and up and down from the reference position, and the measured line-of-sight is moved in the (x, y) and horizontal directions. Let c be the translation component and d be the translation component in the vertical direction, and the line of sight after calibration (x ′, y ′) is expressed by equation (2).

  The enlargement / reduction is a shift caused by the enlargement / reduction of the distribution of the line-of-sight data when the head position moves back and forth from the reference position. If the enlargement / reduction ratio is α, the line of sight after calibration (x ′, y ′) is expressed by equation (3).

  Rotation is a shift caused by a change in the relative position between the driver and the camera 12 when the vehicle is tilted to the left and right, and when the rotation angle is ψ, the line of sight after calibration (x ′, y ′) is (4) It is expressed by a formula.

  By combining these equations (2) to (4), a matrix H shown in equation (5) called Helmart similarity transformation consisting of calibration parameters (a, b, c, d) is calculated.

  Assuming that the calculated gaze position of the rearview mirror is (xb, yb) and the gaze position of the right side mirror is (xr, yr), the known gaze position (x′b, y′b) of the reference rearview mirror , And the known gaze position (x′r, y′r) of the right side mirror are expressed by the equation (6).

  The calibration parameters (a, b, c, d) of the matrix H are calculated from the equation (6), and the process is terminated. The measured line-of-sight data (φ, θ) is calculated and calibrated by the equation (7) using the calibration parameter (matrix H), and calibrated line-of-sight data (φ ′, θ ′) is obtained.

  In addition, if the computer 16 for executing the above processing is described by function blocks divided for each function realizing means determined based on hardware and software, as shown in FIG. An image acquisition unit 34 to be acquired, a line-of-sight data calculation unit 36 that calculates line-of-sight data based on image data acquired by the image acquisition unit, an operation data acquisition unit 38 that acquires operation data detected by the operation sensor 14, and a line-of-sight data calculation Specific operation data from among the line-of-sight data calculated by the unit and the data storage unit 40 that stores the operation data acquired by the operation data acquisition unit in association with time series, and the line-of-sight data stored in the data storage unit And a line-of-sight data extraction unit 42 for extracting line-of-sight data distributed in a predetermined area, and based on the distribution of the extracted line-of-sight data A visual position is calculated and expressed by a configuration including a correction coefficient calculation unit 44 that calculates a correction coefficient for correcting the line-of-sight data so that the gaze position matches a predetermined gaze position for a specific operation. Can do.

  As described above, in the present embodiment, since the line-of-sight data from the predetermined time before the start time of the specific operation to the end time is used and the line-of-sight data distributed in the predetermined region is used, the line-of-sight data with high reliability is used. Thus, the correction coefficient can be calculated, and the accuracy of eye gaze measurement can be improved.

  In the present embodiment, the case where the calibration parameters are calculated using the positions of the rearview mirror and the side mirror has been described. However, the calibration parameters including the front gaze position calculated in step 114 are calculated. It may be.

  In the present embodiment, the specific operation is changed from the left lane to the right lane, and the specific gaze position is the rearview mirror and the right side mirror. It is sufficient if the gaze position can be estimated. For example, the specific operation is changed from the right lane to the left lane, the specific gaze position is set to the rear view mirror and the left side mirror, or the specific operation is performed with an accelerator pedal. A specific gaze position may be used as a speedometer when stepping over a value.

  Further, in the present embodiment, as a method of extracting stationary line-of-sight data, a case has been described in which it is determined whether or not the maximum value of the Euclidean distance of line-of-sight data included in a predetermined section is equal to or less than a predetermined value. If, for example, a variance value of M gaze data included in a predetermined section is calculated and the value is equal to or less than a predetermined value, the gaze data in that section is stopped. It may be determined as the line-of-sight data. In addition, when using a method in which the reliability of the line-of-sight data is given at the time of line-of-sight measurement, the line-of-sight data having a reliability higher than a predetermined position may be used.

  Further, in the present embodiment, the case where the line-of-sight measuring device of the present invention is mounted on a vehicle and the driver's line-of-sight is measured has been described. For example, the line-of-sight measuring device of the present invention is provided in an airplane cockpit and It is also possible to measure the line of sight and set the operating devices and instruments to be watched in response to the pilot's operation as known gaze positions.

  In addition to the operation of such a vehicle, the line-of-sight measurement device of the present invention can be applied. As another embodiment, a case where the present invention is applied to a game machine will be described. The configuration is the same as in the above embodiment.

  When an icon corresponding to the type or mode is displayed on the screen in order to prompt the user to select a game type or mode, the gaze position at the reference time is determined because the icon display position is known. When the user operates the controller or the like to select a desired icon, the user gazes at the icon. Therefore, the selection signal detected when the icon is selected is used as operation data, and is stored in association with the line-of-sight data when the icon is selected. Based on the stored line-of-sight data and operation data, the correction coefficient can be calculated by the same processing as in the above embodiment.

  For example, the capturing of image data is started at the timing when the power of the game device is turned on and the line-of-sight data is calculated. When the icon for selecting a racing game is displayed at the right end of the screen, the icon for selecting a shooting game is displayed at the left end of the screen, and the icon for selecting a baseball game is displayed at the center of the screen, a signal indicating the selection result is operated. Import as data. When the calibration timing is reached, the operation data for selecting the baseball game is searched from the operation data, and the line-of-sight data from the predetermined time Δt before the time t1 when the operation data is acquired to the predetermined time t2 is extracted. t2 can be the time when the next operation data is acquired. Then, the correction coefficient is calculated based on the gaze position calculated from the extracted line-of-sight data and the known gaze position of the icon for selecting the baseball game. Similarly, the operation data for selecting the racing game and the known gaze position of the icon for selecting the racing game may be used, or the operation data for selecting the shooting game and the known gaze position of the icon for selecting the shooting game may be used. May be.

  In such a case, there is a high possibility that the user is gazing at a specific position before performing a specific operation, so it is more possible to extract line-of-sight data from a predetermined time before the start of the specific operation. Appropriate line-of-sight data can be extracted effectively.

It is a block diagram which shows the outline of the visual line measuring apparatus 10 which concerns on this Embodiment. It is a flowchart which shows the processing routine of the correction coefficient calculation program in this Embodiment. It is an example of the figure which plotted gaze data (horizontal direction (phi), vertical direction (theta)), (A) What plotted all the accumulated gaze data, (B) What plotted gaze data corresponding to specific operation data (C) Gaze data corresponding to specific operation data and plotting gaze data distributed in a predetermined area are shown. It is a block diagram which shows the functional structure of the gaze measuring apparatus 10 which concerns on this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Eye-gaze measuring device 12 Camera 14 Operation sensor 16 Computer 34 Image acquisition part 36 Eye-gaze data calculation part 38 Operation data acquisition part 40 Data storage part 42 Eye-gaze data extraction part 44 Correction coefficient calculation part

Claims (4)

Line-of-sight data calculating means for calculating line-of-sight data indicating the line-of-sight direction of the operator based on image data of an eye area obtained by photographing an operator who operates the device;
Obtaining means for obtaining operation data when the operator operates the device;
Storage means for storing line-of-sight data calculated by the line-of-sight data calculation means and operation data acquired by the acquisition means in association with each other in time series;
Extraction means for extracting line-of-sight data from a predetermined time before the start time of a specific operation to the end time of the specific operation for the device from which the operation data stored in the storage means has been acquired from line-of-sight data stored in the storage means; ,
The calculation means such that the gaze position of the operator obtained based on the distribution of the line-of-sight data extracted by the extraction means coincides with a gaze position predetermined as a position to gaze at the specific operation. Correction coefficient calculating means for calculating a correction coefficient for correcting the line-of-sight data calculated in
Eye gaze measuring device including   The extraction means is line-of-sight data from a predetermined time before the start time of a specific operation to the end time of the specific operation for the device from which the operation data stored in the storage means has been acquired, and a line of sight distributed in a predetermined area The line-of-sight measurement apparatus according to claim 1, wherein data is extracted from line-of-sight data stored in the storage unit. Computer
Line-of-sight data calculating means for calculating line-of-sight data indicating the line-of-sight direction of the operator based on image data of an eye area obtained by photographing an operator who operates the device;
Obtaining means for obtaining operation data when the operator operates the device;
A specific operation for the device from which the operation data stored in the storage unit that stores the line-of-sight data calculated by the line-of-sight data calculation unit and the operation data acquired by the acquisition unit in association with each other in time series is acquired. Extraction means for extracting line-of-sight data from a predetermined time before the start time to the end time from line-of-sight data stored in the storage means;
The calculation means such that the gaze position of the operator obtained based on the distribution of the line-of-sight data extracted by the extraction means coincides with a gaze position predetermined as a position to gaze at the specific operation. Correction coefficient calculating means for calculating a correction coefficient for correcting the line-of-sight data calculated in
Gaze measurement program to make it function.   A line-of-sight measurement program for causing a computer to function as each means constituting the line-of-sight measurement device according to claim 2.

Images