[go: up one dir, main page]

WO2025216110A1 - Information processing device, information processing method, and program - Google Patents

Information processing device, information processing method, and program

Info

Publication number
WO2025216110A1
WO2025216110A1 PCT/JP2025/013086 JP2025013086W WO2025216110A1 WO 2025216110 A1 WO2025216110 A1 WO 2025216110A1 JP 2025013086 W JP2025013086 W JP 2025013086W WO 2025216110 A1 WO2025216110 A1 WO 2025216110A1
Authority
WO
WIPO (PCT)
Prior art keywords
head pose
correlation
rotation
information processing
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2025/013086
Other languages
French (fr)
Japanese (ja)
Inventor
健 西田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Publication of WO2025216110A1 publication Critical patent/WO2025216110A1/en
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Definitions

  • the present invention relates to an information processing device, an information processing method, and a program.
  • gaze estimation technology is applied to reduce the load on rendering processing (Foveated Rendering).
  • further improvements in the accuracy of gaze estimation are required to realize gaze-based UIs (User Interfaces), express DoF (Depth of Field), and correct display distortion caused by eye movements (Swim Correction).
  • Patent Document 1 discloses technology that corrects the positional relationship between the optical axis and visual axis by estimating the direction of gravity using an acceleration sensor and estimating the rotation of the eyeball from the direction of gravity.
  • Patent Document 2 discloses technology that estimates the rotation of the eyeball from the iris pattern.
  • detecting iris patterns places a high processing load on the system, making it difficult to operate constantly in an HMD.
  • this disclosure proposes an information processing device, information processing method, and program that are capable of highly accurate gaze estimation.
  • the present disclosure provides an information processing device having a headpose-rotation correlation unit that acquires the correlation between eye rotation and head pose as a headpose-rotation correlation, a headpose estimation unit that acquires the head pose, and an optical axis-visual axis correlation unit that applies the head pose to the headpose-rotation correlation to estimate the eye rotation and corrects the visual axis of the eye based on the estimated eye rotation.
  • the present disclosure also provides an information processing method in which the information processing of the information processing device is executed by a computer, and a program that causes a computer to implement the information processing of the information processing device.
  • FIG. 10 is an explanatory diagram of gaze estimation taking into account eye rotation.
  • FIG. 10 is an explanatory diagram of gaze estimation taking into account eye rotation.
  • FIG. 1 is a diagram illustrating an example of a configuration of a display system.
  • FIG. 1 is a diagram showing a conventional display system serving as a comparative example.
  • FIG. 10 is a diagram showing an example of a calibration image for investigating the correlation between eye rotation and head pose.
  • FIG. 10 is a diagram showing an example of a calibration image for examining the correlation between the optical axis and the visual axis.
  • FIG. 10 is a diagram showing an example of a processing flow for detecting head pose-rotation correlation by a calibration operation before viewing content.
  • FIG. 10 is a diagram illustrating an example of a processing flow for estimating a line of sight when viewing content.
  • FIG. 10 is a diagram illustrating another configuration example of the display system.
  • FIG. 2 illustrates an example of a hardware configuration of an information processing device.
  • Gaze estimation taking into account eye rotation [2. System configuration example] [3. Display example of calibration image] 4. Calibration Processing Flow 5. Gaze Estimation Processing Flow 6. Sequential update of headpose-rotation correlation [7. Hardware Configuration Example] 8. Effects
  • Gaze estimation taking into account eye rotation 1 and 2 are explanatory diagrams of gaze estimation taking into account eye rotation.
  • Eye tracking is used in a variety of fields. Eye tracking is a technology that tracks what a user US is looking at in real time based on eye movements.
  • the eye tracker detects the optical axis LA of the eyeball EB based on an image of the eye (ocular image).
  • the optical axis LA is the axis that passes through the center of the cornea CR and the center of the pupil PU, but the line of sight does not necessarily coincide with the optical axis LA of the eyeball EB.
  • the distribution of photoreceptors on the retina is not uniform, and the line of sight is the line connecting the area with a high density of photoreceptors (the depression in the center of the macula MC: the macular fovea centralis) and the nodal point (the central posterior surface of the lens).
  • the axis along which the line of sight passes is called the visual axis VA.
  • the visual axis VA cannot be determined directly from the eye image.
  • the visual axis VA is tilted at about 5° with respect to the optical axis LA.
  • the offset between the visual axis VA and the optical axis LA varies from person to person, and this individual difference must be adjusted through calibration.
  • the visual axis VA (line of sight) is estimated from the optical axis LA based on the calibration information.
  • the eyeball EB rotates.
  • the three-dimensional positional relationship between the optical axis LA and the visual axis VA changes.
  • the correlation between eyeball rotation and head pose is obtained in advance, and the visual axis VA is corrected based on the eyeball rotation estimated from the head pose.
  • FIG. 3 is a diagram showing an example of the configuration of the display system 1. As shown in FIG.
  • the display system 1 has an information processing device 10 and an HMD 20.
  • the HMD 20 presents 3D images to the user US who is wearing the HMD.
  • the information processing device 10 detects the user US's line of sight based on the eye movements of the user US and controls the display according to the line of sight.
  • Figure 3 selectively shows configuration related to line of sight estimation. Configuration normally used only for processing content (movies, games, etc.) is omitted from the illustration.
  • the HMD 20 has an IMU (Inertial Measurement Unit) 21, an external camera 22, a display 23, and an eye camera 24.
  • the external camera 22 captures images of the surroundings of the HMD 20. Sensing information from the IMU 21 and external camera 22 is used to estimate the head pose of the user US.
  • the display 23 displays an image for calibration.
  • the eye camera 24 captures images of the eyes of the user US and acquires eye images.
  • the information processing device 10 has a SLAM signal processing unit 11, a head pose estimation unit 12, a head pose-rotation correlation unit 13, a target display unit 14, an iris detection unit 15, a rotation detection unit 16, an optical axis estimation unit 17, an optical axis-visual axis correlation unit 18, and a visual axis estimation unit 19.
  • the SLAM signal processing unit 11 estimates the user's position from sensing information from the IMU 21 and external camera 22. Self-position estimation can be performed using SLAM (Simultaneous Localization and Mapping) technology.
  • the head pose estimation unit 12 acquires the head pose of the user US based on the estimated self-position information.
  • the headpose-rotation correlation unit 13 obtains the correlation between eye rotation and head pose as the headpose-rotation correlation.
  • the headpose-rotation correlation is determined through a prior calibration process by the user US.
  • the headpose-rotation correlation unit 13 applies the headpose estimated by the headpose estimation unit 12 to the headpose-rotation correlation to obtain the eye rotation (direction and angle of rotation) corresponding to the head pose.
  • the headpose-rotation correlation unit 13 stores the correlation between eye rotation and headpose acquired for the user of the calibration information (the user US whose headpose is to be acquired when viewing content).
  • the headpose-rotation correlation may also be obtained by aggregating headpose and eye rotation data collected from multiple people.
  • the iris detection unit 15 extracts the iris from the eye image acquired from the eye camera 24.
  • the rotation detection unit 16 detects the rotation of the eyeball EB based on the iris image.
  • Information on eyeball rotation is output to the headpose-rotation correlation unit 13.
  • the headpose-rotation correlation unit 13 links the eyeball rotation information acquired from the rotation detection unit 16 with the headpose information acquired from the headpose estimation unit 12. In this way, the headpose-rotation correlation is acquired.
  • the optical axis estimation unit 17 acquires the optical axis LA of the eyeball EB based on the eye image acquired from the eye camera 24.
  • the optical axis-visual axis correlation unit 18 acquires the correlation between the optical axis LA of the eyeball EB and the visual axis VA as the optical axis-visual axis correlation.
  • the optical axis-visual axis correlation is determined through a prior calibration operation by the user US.
  • the optical axis-visual axis correlation unit 18 stores the correlation between the optical axis LA and the visual axis VA acquired for the user of the calibration information (the user US from whom the optical axis LA is acquired when viewing content).
  • the optical axis-visual axis correlation may also be acquired by aggregating data on the optical axis LA and visual axis VA collected from multiple people.
  • the target display unit 14 generates an image (calibration image) to be used in the calibration process and displays it on the display 23.
  • the calibration image includes a target TG (see Figures 5 and 6) that is the gaze target.
  • the target display unit 14 displays the target TG that the user US is looking at on the display 23.
  • the target display unit 14 moves the position of the target TG to achieve natural line of sight and head movements required during calibration.
  • the target display unit 14 generates calibration images, including an image for examining the correlation between eye rotation and head pose (see Figure 5) and an image for examining the correlation between the optical axis LA and the visual axis VA (see Figure 6).
  • the calibration image encourages the eyes to follow the target TG, which is the object of gaze, and thereby encourages fluctuations in the eyeballs EB and head pose.
  • the rotation detection unit 16 detects the eyeball rotation of the user US that accompanies the movement of the target TG.
  • the head pose estimation unit 12 acquires the head pose of the user US that fluctuates as the target TG moves.
  • the head pose-rotation correlation unit 13 acquires the correlation between the eyeball rotation and head pose that accompanies the movement of the target TG as the head pose-rotation correlation.
  • the correlation between the optical axis LA and the visual axis VA can be obtained based on the position of the target TG and information about the optical axis LA.
  • the optical axis-visual axis correlation unit 18 estimates the visual axis VA of the eyeball EB based on the position of the target TG obtained from the target display unit 14.
  • the optical axis-visual axis correlation unit 18 obtains the optical axis-visual axis correlation by linking the estimated visual axis VA with the optical axis LA obtained from the optical axis estimation unit 17.
  • the optical axis-visual axis correlation unit 18 obtains the eye rotation corresponding to the headpose from the headpose-rotation correlation unit 13.
  • the eye rotation is obtained by applying the headpose estimated by the headpose estimation unit 12 to the headpose-rotation correlation.
  • the optical axis-visual axis correlation unit 18 estimates the visual axis VA by applying the optical axis LA obtained from the optical axis estimation unit 17 to the optical axis-visual axis correlation.
  • the optical axis-visual axis correlation unit 18 corrects the visual axis VA of the eye EB based on the eye rotation obtained from the headpose-rotation correlation unit 13.
  • the visual axis estimation unit 19 outputs the visual axis VA, whose orientation has been corrected by the correction, as the visual axis VA corresponding to the headpose.
  • Fig. 4 is a diagram showing a conventional display system 1C as a comparative example.
  • Fig. 4 selectively depicts configurations related to gaze estimation, and omits configurations used only for processing regular content (movies, games, etc.).
  • gaze estimation is performed based only on the correlation between the optical axis LA and the visual axis VA.
  • the target display unit 14C displays only an image for examining the correlation between the optical axis LA and the visual axis VA. Since the correlation between head pose and eye rotation and the effect of eye rotation on the visual axis VA are not taken into consideration, appropriate gaze estimation is not performed when the head moves. In the method of the present disclosure shown in FIG. 3, the relationship between head pose and the visual axis VA is taken into consideration, and therefore appropriate gaze estimation is performed even when the head moves.
  • Figure 5 shows an example of a calibration image for investigating the correlation between eye rotation and head pose.
  • a spherical object floating in virtual space is presented as the target TG.
  • the head pose estimation unit 12 tracks the head movement of the user US who is following the target TG.
  • the target display unit 14 guides the head movement by varying the display position of the target TG.
  • the target display unit 14 displays the target TG in a position that induces a change in head pose in order to achieve natural head movement.
  • the target display unit 14 displays an obstruction OB in front of the target TG, encouraging the head pose to be changed as if peering at the target TG behind the obstruction OB.
  • the target display unit 14 can also notify the user by voice or text of a message encouraging the head pose to be changed.
  • the calibration process is performed as a preliminary step before viewing regular content (movies, games, etc.). It is preferable that the subject of the calibration process be the same user US as the viewer of the regular content (the user US whose head pose is to be acquired when viewing the regular content). This allows for appropriate correction of the visual axis VA, taking into account the individual differences of the user US.
  • the visual axis VA can also be corrected based on standard correlation data obtained from a large number of subjects.
  • the headpose-rotation correlation unit 13 can store the correlation between average eye rotation and headpose based on data from multiple subjects as the headpose-rotation correlation. In this case, generally good gaze estimation can be performed without prior calibration.
  • FIG. 6 shows an example of a calibration image for investigating the correlation between the optical axis LA and the visual axis VA.
  • one point is selected from multiple radially arranged points, and the selected point is presented as the target TG by, for example, lighting up.
  • the calibration image is presented as an image (headlock image HI) that allows all points to be seen with the head fixed.
  • the iris detection unit 15 tracks the movement (eye rotation) of the eyeball EB of the user US as it follows the target TG.
  • the target display unit 14 guides the movement of the eyeball EB by varying the display position of the target TG.
  • the iris detection unit 15 estimates eye rotation based on the iris pattern. Detecting the iris pattern requires high processing power, making it difficult to perform constantly when viewing normal content. However, if the iris pattern detection process is limited to calibration work, processing load issues are unlikely to arise. When viewing normal content, the processing load is reduced by performing gaze estimation based on the process of estimating the optical axis LA and the process of converting the optical axis LA to the visual axis VA.
  • FIG. 7 is a diagram showing an example of a processing flow for detecting head pose-rotation correlation through a calibration operation before viewing content.
  • the target display unit 14 displays a calibration image including the target TG on the display 23 (step S1).
  • the SLAM signal processing unit 11 estimates the self-position of the user US during the calibration operation based on sensor information acquired from the IMU 21 and external camera 22.
  • the head pose estimation unit 12 acquires the head pose of the user US during the calibration operation based on the estimated self-position (step S2).
  • the eye camera 24 acquires an image of the eye of the user US following the target TG.
  • the iris detection unit 15 detects the iris from the eye image.
  • the rotation detection unit 16 detects eye rotation based on the tilt of the iris pattern (step S3).
  • the head pose-rotation correlation unit 13 links the head pose acquired from the head pose estimation unit 12 with the eye rotation acquired from the rotation detection unit 16, and records the correspondence between the two as a head pose-rotation correlation (step S4). Note that steps S2 and S3 are performed in parallel, and either may be performed first.
  • FIG. 8 is a diagram showing an example of a processing flow of gaze estimation during content viewing.
  • the SLAM signal processing unit 11 estimates the user US's self-position while viewing content based on sensor information acquired from the IMU 21 and external camera 22.
  • the head pose estimation unit 12 acquires the head pose of the user US while viewing content based on the estimated self-position (step S11).
  • the optical axis-visual axis correlation unit 18 applies the acquired head pose to the head pose-rotation correlation to estimate the eye rotation of the user US (step S12).
  • the eye camera 24 acquires an image of the eye of the user US while viewing content.
  • the optical axis estimation unit 17 estimates the optical axis LA of the eyeball EB based on the eye image (step S13).
  • the optical axis-visual axis correlation unit 18 applies the estimated optical axis LA to the optical axis-visual axis correlation to estimate the visual axis VA of the user US while viewing content (step S14).
  • the optical axis-visual axis correlation unit 18 corrects the direction of the visual axis VA based on the eye rotation estimated from the head pose (step S15). Note that steps S11-S12 and steps S13-S14 are performed in parallel, and either may be performed first.
  • Sequential update of headpose-rotation correlation 9 is a diagram showing another example of the configuration of a display system. The following description will focus on the differences from the display system 1 shown in FIG.
  • the display system omits the target display unit 14 shown in Figure 3.
  • no dedicated image is generated for the calibration process.
  • an image of the content being viewed is extracted based on some kind of trigger, and the extracted image is used as the calibration image.
  • Triggers can be set arbitrarily by the system developer. For example, a trigger can be set when a scene containing a clear target of gaze (attractive area), such as a scene of a glowing object flying in the dark, is detected.
  • the headpose-rotation correlation unit 13 sequentially updates the headpose-rotation correlation based on eye rotation and headpose data of the user US acquired intermittently based on preset triggers.
  • the information processing device 30 may include a scene detection unit.
  • the scene detection unit detects a video scene including an attention region as a peculiar scene.
  • an attention region is a display area of an object that is likely to attract the attention of the user US and is likely to induce head movement of the user US by moving over a wide area of the screen. In a scene in which a glowing object is flying in the dark, the glowing flying object becomes the attention region.
  • the rotation detection unit 16 detects eye rotation of the user US accompanying movement of the eye region when a peculiar scene is detected.
  • the head pose estimation unit 12 acquires the head pose of the user US, which changes as the eye region moves.
  • the head pose-rotation correlation unit 13 updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the eye region and the head pose.
  • FIG. 10 is a diagram illustrating an example of the hardware configuration of the information processing device 10.
  • Computer 1000 has a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.
  • CPU Central Processing Unit
  • RAM Random Access Memory
  • ROM Read Only Memory
  • HDD Hard Disk Drive
  • CPU 1100 operates based on programs (program data 1450) stored in ROM 1300 or HDD 1400, and controls each component. For example, CPU 1100 loads programs stored in ROM 1300 or HDD 1400 into RAM 1200 and executes processing corresponding to the various programs.
  • BIOS Basic Input Output System
  • HDD 1400 is a computer-readable, non-transitory recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs.
  • HDD 1400 is a recording medium that records an information processing program according to an embodiment, which is an example of program data 1450.
  • the communication interface 1500 is an interface that allows the computer 1000 to connect to an external network 1550 (such as the Internet).
  • an external network 1550 such as the Internet
  • the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.
  • the input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000.
  • the CPU 1100 receives data from input devices such as a keyboard or mouse via the input/output interface 1600.
  • the CPU 1100 also transmits data to output devices such as a display device, speaker, or printer via the input/output interface 1600.
  • the input/output interface 1600 may also function as a media interface that reads programs recorded on a specified recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.
  • HDD 1400 also stores the information processing program, various models, and various data according to the present disclosure. While CPU 1100 reads and executes program data 1450 from HDD 1400, as another example, it may also obtain these programs from other devices via external network 1550.
  • the information processing device 10 has a head pose-rotation correlation unit 13, a head pose estimation unit 12, and an optical axis-visual axis correlation unit 18.
  • the head pose-rotation correlation unit 13 obtains the correlation between eye rotation and head pose as a head pose-rotation correlation.
  • the head pose estimation unit 12 obtains the head pose.
  • the optical axis-visual axis correlation unit 18 corrects the visual axis VA of the eye EB based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation.
  • the processing of the information processing device 10 is executed by a computer 1000.
  • a program disclosed herein causes the computer 1000 to realize the processing of the information processing device 10.
  • the visual axis VA is corrected while appropriately taking into account the rotation of the eyeball EB. This enables highly accurate gaze estimation.
  • the head pose-rotation correlation unit 13 stores the correlation between eye rotation and head pose acquired for the user US whose head pose is to be acquired.
  • This configuration allows for appropriate correction of the visual axis VA, taking into account individual differences among users US.
  • the information processing device 10 has a target display unit 14 and a rotation detection unit 16.
  • the target display unit 14 displays a target TG that the user US is focusing on on the display 23.
  • the rotation detection unit 16 detects eye rotation of the user US that accompanies movement of the target TG.
  • the head pose estimation unit 12 acquires the head pose of the user US that changes as the target TG moves.
  • the head pose-rotation correlation unit 13 acquires the correlation between the eye rotation and head pose that accompanies movement of the target TG as the head pose-rotation correlation.
  • the target display unit 14 displays a message encouraging the user to change their head pose.
  • This configuration ensures that head pose changes are implemented reliably.
  • the target display unit 14 displays the target TG in a position that induces a change in head pose.
  • This configuration ensures that head pose changes are implemented reliably.
  • the target display unit 14 displays an obstruction OB in front of the target TG. This encourages the head pose movement to look at the target TG behind the obstruction OB.
  • This configuration ensures that head pose changes are implemented reliably.
  • the headpose-rotation correlation unit 13 sequentially updates the headpose-rotation correlation based on eye rotation and headpose data of the user US acquired intermittently based on preset triggers.
  • This configuration allows for accurate headpose-rotation correlation that reflects individual differences among users US.
  • the information processing device 10 has a scene detection unit.
  • the scene detection unit detects video scenes including an attention region as peculiar scenes.
  • the rotation detection unit 16 triggered by the detection of a peculiar scene, detects eye rotation of the user US accompanying movement of the attention region.
  • the head pose estimation unit 12 acquires the head pose of the user US, which fluctuates as the attention region moves.
  • the head pose-rotation correlation unit 13 updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the attention region and the head pose.
  • head pose-rotation correlation can be acquired naturally during viewing of normal content (movies, games, etc.).
  • the headpose-rotation correlation unit 13 stores the correlation between average eye rotation and headpose based on data from multiple subjects as the headpose-rotation correlation.
  • This configuration allows for generally good gaze estimation while omitting prior calibration.
  • the information processing device 10 has an optical axis estimation unit 17.
  • the optical axis estimation unit 17 acquires the optical axis LA of the eyeball EB.
  • the optical axis-visual axis correlation unit 18 acquires the correlation between the optical axis LA and the visual axis VA as the optical axis-visual axis correlation.
  • the optical axis-visual axis correlation unit 18 applies the optical axis LA to the optical axis-visual axis correlation to estimate the visual axis VA.
  • the optical axis-visual axis correlation unit 18 corrects the estimated visual axis VA based on eye rotation.
  • the visual axis VA can be estimated with high accuracy from the optical axis LA. This allows for highly accurate gaze estimation.
  • the optical axis-visual axis correlation unit 18 stores the correlation between the optical axis LA and the visual axis VA obtained for the user US for whom the optical axis LA is to be obtained.
  • This configuration enables accurate gaze estimation that reflects individual differences among users US.
  • the present technology can also be configured as follows.
  • a headpose-rotation correlation unit that acquires a correlation between eye rotation and head pose as a headpose-rotation correlation; a head pose estimation unit for acquiring the head pose; an optical axis-visual axis correlation unit that corrects the visual axis of the eyeball based on the eyeball rotation obtained by applying the headpose to the headpose-rotation correlation; An information processing device having the above.
  • the head pose-rotation correlation unit stores the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired.
  • the information processing device according to (1) above.
  • a target display unit that displays a target that the user is paying attention to on a display; a rotation detection unit that detects the eye rotation of the user accompanying the movement of the target; and the head pose estimation unit acquires the head pose of the user that varies in accordance with movement of the target; the headpose-rotation correlation unit acquires a correlation between the eye rotation accompanying the movement of the target and the headpose as the headpose-rotation correlation;
  • the target display unit notifies a message prompting the user to change the head pose.
  • the target display unit displays the target at a position that causes a change in the head pose.
  • the target display unit displays an obstruction in front of the target, and prompts the user to move the head pose in a manner that looks at the target behind the obstruction;
  • the information processing device according to (5) above.
  • the headpose-rotation correlator sequentially updates the headpose-rotation correlation based on data of the eye rotation and the headpose of the user intermittently acquired based on a preset trigger;
  • the information processing device according to any one of (2) to (6) above.
  • a scene detection unit that detects a video scene including an attention region as a peculiar scene
  • a rotation detection unit that detects the eye rotation of the user accompanying the movement of the attraction region, using the detection of the peculiar scene as the trigger
  • the head pose estimation unit acquires the head pose of the user that varies in accordance with the movement of the interest region
  • the head pose-rotation correlation unit updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying the movement of the attention region and the head pose.
  • the headpose-rotation correlation unit stores an average correlation between the eye rotation and the headpose based on data of a plurality of subjects as the headpose-rotation correlation;
  • the information processing device according to (1) above.
  • the optical axis-visual axis correlation unit Obtaining a correlation between the optical axis and the visual axis as an optical axis-visual axis correlation; applying the optical axis to the optical axis-visual axis correlation to estimate the visual axis; correcting the estimated visual axis based on the eye rotation;
  • the information processing device according to any one of (1) to (9) above.
  • the optical axis-visual axis correlation unit stores the correlation between the optical axis and the visual axis acquired for the user from whom the optical axis is to be acquired.
  • the information processing device according to (10) above.
  • a computer-implemented information processing method comprising: (13) storing the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired; The information processing method according to (12) above.
  • the process of acquiring the head pose-rotation correlation includes acquiring a correlation between the eye rotation accompanying the movement of the target and the head pose as the head pose-rotation correlation.
  • the information processing method according to (13) above. notifying a message prompting the head pose change; The information processing method according to (14) above.
  • the target display process displays the target at a position that causes a change in the head pose.
  • the target display process includes displaying an obstruction in front of the target, and prompting the user to move the head pose so as to look at the target behind the obstruction.
  • the process of acquiring the head pose-rotation correlation includes sequentially updating the head pose-rotation correlation based on data of the eye rotation and the head pose of the user acquired intermittently based on a preset trigger.
  • the process of acquiring the head pose-rotation correlation includes storing an average correlation between the eye rotation and the head pose based on data of a plurality of subjects as the head pose-rotation correlation.
  • Head pose estimation unit 13 Head pose-rotation correlation unit 14 Target display unit 16 Rotation detection unit 17 Optical axis estimation unit 18 Optical axis-visual axis correlation unit 23 Display EB Eyeball LA Optical axis OB Obstruction object TG Target US User VA Visual axis

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Ophthalmology & Optometry (AREA)
  • Biophysics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

This information processing device includes a head pose-rotation correlation unit, a head pose estimation unit, and an optical axis-visual axis correlation unit. The head pose-rotation correlation unit acquires a correlation between the eyeball rotation and the head pose as a head pose-rotation correlation. The head pose estimation unit acquires the head pose. The optical axis-visual axis correlation unit applies the head pose to the head pose-rotation correlation so as to estimate the eyeball rotation. The optical axis-visual axis correlation unit corrects the visual axis of the eyeball on the basis of the estimated eyeball rotation.

Description

情報凊理装眮、情報凊理方法およびプログラムInformation processing device, information processing method, and program

　本発明は、情報凊理装眮、情報凊理方法およびプログラムに関する。 The present invention relates to an information processing device, an information processing method, and a program.

　　　などでは、芖線掚定技術が描画凊理負荷䜎枛　に応甚されおいる。しかし、芖線による　の実珟、　　衚珟、県振りによるディスプレむ歪補正補正を行うためには、さらなる芖線掚定の粟床向䞊が必芁である。 In devices such as HMDs (Head Mounted Displays), gaze estimation technology is applied to reduce the load on rendering processing (Foveated Rendering). However, further improvements in the accuracy of gaze estimation are required to realize gaze-based UIs (User Interfaces), express DoF (Depth of Field), and correct display distortion caused by eye movements (Swim Correction).

米囜特蚱出願公開第号明现曞US Patent Application Publication No. 2016/0085299 特開号公報Japanese Patent Application Laid-Open No. 2022-095879

　県の光軞ず芖軞は䞀般に䞀臎しおおらず、たた頭を傟けるず県球は回旋する。県球が回旋するず光軞ず芖軞の関係は回旋に埓っお倉化しおしたう。特蚱文献には、加速床センサを甚いお重力方向を掚定し、重力方向から県球の回旋を掚定するこずによっお、光軞ず芖軞の䜍眮関係を補正する技術が開瀺されおいる。しかし、県球の回旋は重力方向に完党に䞀臎するわけではないので、粟床の高い補正はできない。特蚱文献には、虹圩パタヌンから県球の回旋を掚定する技術が開瀺されおいる。しかし、虹圩パタヌンの怜出は凊理負荷が高く、においお垞時動䜜させるこずは困難である。 The optical axis and visual axis of the eye generally do not coincide, and tilting the head causes the eyeball to rotate. When the eyeball rotates, the relationship between the optical axis and visual axis changes accordingly. Patent Document 1 discloses technology that corrects the positional relationship between the optical axis and visual axis by estimating the direction of gravity using an acceleration sensor and estimating the rotation of the eyeball from the direction of gravity. However, because the rotation of the eyeball does not perfectly coincide with the direction of gravity, highly accurate correction is not possible. Patent Document 2 discloses technology that estimates the rotation of the eyeball from the iris pattern. However, detecting iris patterns places a high processing load on the system, making it difficult to operate constantly in an HMD.

　そこで、本開瀺では、粟床のよい芖線掚定が可胜な情報凊理装眮、情報凊理方法およびプログラムを提案する。 Therefore, this disclosure proposes an information processing device, information processing method, and program that are capable of highly accurate gaze estimation.

　本開瀺によれば、県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗するヘッドポヌズ回旋盞関郚ず、前蚘ヘッドポヌズを取埗するヘッドポヌズ掚定郚ず、前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお前蚘県球回旋を掚定し、掚定された前蚘県球回旋に基づいお県球の芖軞を補正する光軞芖軞盞関郚ず、を有する情報凊理装眮が提䟛される。たた、本開瀺によれば、前蚘情報凊理装眮の情報凊理がコンピュヌタにより実行される情報凊理方法、ならびに、前蚘情報凊理装眮の情報凊理をコンピュヌタに実珟させるプログラムが提䟛される。 The present disclosure provides an information processing device having a headpose-rotation correlation unit that acquires the correlation between eye rotation and head pose as a headpose-rotation correlation, a headpose estimation unit that acquires the head pose, and an optical axis-visual axis correlation unit that applies the head pose to the headpose-rotation correlation to estimate the eye rotation and corrects the visual axis of the eye based on the estimated eye rotation. The present disclosure also provides an information processing method in which the information processing of the information processing device is executed by a computer, and a program that causes a computer to implement the information processing of the information processing device.

県球回旋を考慮した芖線掚定の説明図である。FIG. 10 is an explanatory diagram of gaze estimation taking into account eye rotation. 県球回旋を考慮した芖線掚定の説明図である。FIG. 10 is an explanatory diagram of gaze estimation taking into account eye rotation. 衚瀺システムの構成の䞀䟋を瀺す図である。FIG. 1 is a diagram illustrating an example of a configuration of a display system. 比范䟋ずなる埓来の衚瀺システムを瀺す図である。FIG. 1 is a diagram showing a conventional display system serving as a comparative example. 県球回旋ずヘッドポヌズずの盞関を調べるためのキャリブレヌション甚画像の䞀䟋を瀺す図である。FIG. 10 is a diagram showing an example of a calibration image for investigating the correlation between eye rotation and head pose. 光軞ず芖軞ずの盞関を調べるためのキャリブレヌション甚画像の䞀䟋を瀺す図である。FIG. 10 is a diagram showing an example of a calibration image for examining the correlation between the optical axis and the visual axis. コンテンツ芖聎前のキャリブレヌション䜜業によっおヘッドポヌズ回旋盞関を怜出するための凊理フロヌの䞀䟋を瀺す図である。FIG. 10 is a diagram showing an example of a processing flow for detecting head pose-rotation correlation by a calibration operation before viewing content. コンテンツ芖聎時の芖線掚定の凊理フロヌの䞀䟋を瀺す図である。FIG. 10 is a diagram illustrating an example of a processing flow for estimating a line of sight when viewing content. 衚瀺システムの他の構成䟋を瀺す図である。FIG. 10 is a diagram illustrating another configuration example of the display system. 情報凊理装眮のハヌドりェア構成の䞀䟋を瀺す図である。FIG. 2 illustrates an example of a hardware configuration of an information processing device.

　以䞋に、本開瀺の実斜圢態に぀いお図面に基づいお詳现に説明する。以䞋の各実斜圢態においお、同䞀の郚䜍には同䞀の笊号を付するこずにより重耇する説明を省略する。 Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, identical components will be designated by the same reference numerals, and duplicate descriptions will be omitted.

　なお、説明は以䞋の順序で行われる。
県球回旋を考慮した芖線掚定
システム構成䟋
キャリブレヌション甚画像の衚瀺䟋
キャリブレヌションの凊理フロヌ
芖線掚定の凊理フロヌ
ヘッドポヌズ回旋盞関の逐次曎新
ハヌドりェア構成䟋
効果 The explanation will be given in the following order.
[1. Gaze estimation taking into account eye rotation]
[2. System configuration example]
[3. Display example of calibration image]
4. Calibration Processing Flow
5. Gaze Estimation Processing Flow
6. Sequential update of headpose-rotation correlation
[7. Hardware Configuration Example]
8. Effects

県球回旋を考慮した芖線掚定
　図および図は、県球回旋を考慮した芖線掚定の説明図である。 [1. Gaze estimation taking into account eye rotation]
1 and 2 are explanatory diagrams of gaze estimation taking into account eye rotation.

　アむトラッキングは様々な分野で利甚されおいる。アむトラッキングは、県の動きに基づいお、ナヌザが䜕を芋おいるのかをリアルタむムに远跡する技術である。アむトラッカは、県の画像県画像に基づいお県球の光軞を怜出する。光軞は角膜の䞭心ず瞳孔の䞭心を通る軞を意味するが、芖線は県球の光軞ずは必ずしも䞀臎しない。網膜䞊の芖现胞の分垃は均䞀ではなく、芖现胞の密床が高い郚䜍黄斑の䞭倮郚の陥没黄斑䞭心窩ず節点氎晶䜓䞭倮埌面ずを結ぶ線が芖線ずなる。 Eye tracking is used in a variety of fields. Eye tracking is a technology that tracks what a user US is looking at in real time based on eye movements. The eye tracker detects the optical axis LA of the eyeball EB based on an image of the eye (ocular image). The optical axis LA is the axis that passes through the center of the cornea CR and the center of the pupil PU, but the line of sight does not necessarily coincide with the optical axis LA of the eyeball EB. The distribution of photoreceptors on the retina is not uniform, and the line of sight is the line connecting the area with a high density of photoreceptors (the depression in the center of the macula MC: the macular fovea centralis) and the nodal point (the central posterior surface of the lens).

　芖線が通る軞は芖軞ず呌ばれる。芖軞は県画像からは盎接求められない。芖軞は光軞に察しお°皋床傟いおいる。芖軞ず光軞ずの間のずれオフセットには個人差があり、この個人差はキャリブレヌションによっお調敎する必芁がある。芖軞芖線はキャリブレヌション情報に基づいお光軞から掚定される。 The axis along which the line of sight passes is called the visual axis VA. The visual axis VA cannot be determined directly from the eye image. The visual axis VA is tilted at about 5° with respect to the optical axis LA. The offset between the visual axis VA and the optical axis LA varies from person to person, and this individual difference must be adjusted through calibration. The visual axis VA (line of sight) is estimated from the optical axis LA based on the calibration information.

　頭を傟けるず県球は回旋する。県球が回旋するず、光軞ず芖軞の次元的な䜍眮関係が倉化する。粟床のよい芖線掚定を行うためには、県球の回線を考慮した芖軞の補正が必芁ずなる。本開瀺では、県球回旋ずヘッドポヌズずの間の盞関を事前に取埗し、ヘッドポヌズから掚定される県球回旋に基づいお芖軞の補正を行う。以䞋、本手法を実珟するための衚瀺システムに぀いお具䜓的に説明する。 When the head is tilted, the eyeball EB rotates. When the eyeball EB rotates, the three-dimensional positional relationship between the optical axis LA and the visual axis VA changes. To perform accurate gaze estimation, it is necessary to correct the visual axis VA taking into account the line of the eyeball EB. In this disclosure, the correlation between eyeball rotation and head pose is obtained in advance, and the visual axis VA is corrected based on the eyeball rotation estimated from the head pose. Below, a display system for realizing this method is described in detail.

システム構成䟋
　図は、衚瀺システムの構成の䞀䟋を瀺す図である。 [2. System configuration example]
FIG. 3 is a diagram showing an example of the configuration of the display system 1. As shown in FIG.

　衚瀺システムは、情報凊理装眮およびを有する。は、装着者であるナヌザに映像を提瀺する。情報凊理装眮は、ナヌザの県の動きに基づいおナヌザの芖線を怜出し、芖線に応じた衚瀺制埡を行う。図には、芖線掚定に関わる構成が遞択的に蚘茉されおいる。通垞コンテンツ映画、ゲヌムなどの凊理のみに甚いられる構成の図瀺は省略されおいる。 The display system 1 has an information processing device 10 and an HMD 20. The HMD 20 presents 3D images to the user US who is wearing the HMD. The information processing device 10 detects the user US's line of sight based on the eye movements of the user US and controls the display according to the line of sight. Figure 3 selectively shows configuration related to line of sight estimation. Configuration normally used only for processing content (movies, games, etc.) is omitted from the illustration.

　は、　　、倖カメラ、ディスプレむおよびアむカメラを有する。倖カメラは、の呚囲を撮圱する。ず倖カメラのセンシング情報は、ナヌザのヘッドポヌズの掚定凊理に甚いられる。ディスプレむは、キャリブレヌション甚画像を衚瀺する。アむカメラは、ナヌザの県を撮圱し、県画像を取埗する。 The HMD 20 has an IMU (Inertial Measurement Unit) 21, an external camera 22, a display 23, and an eye camera 24. The external camera 22 captures images of the surroundings of the HMD 20. Sensing information from the IMU 21 and external camera 22 is used to estimate the head pose of the user US. The display 23 displays an image for calibration. The eye camera 24 captures images of the eyes of the user US and acquires eye images.

　情報凊理装眮は、信号凊理郚、ヘッドポヌズ掚定郚、ヘッドポヌズ回旋盞関郚、タヌゲット衚瀺郚、虹圩怜出郚、回旋怜出郚、光軞掚定郚、光軞芖軞盞関郚および芖軞掚定郚を有する。 The information processing device 10 has a SLAM signal processing unit 11, a head pose estimation unit 12, a head pose-rotation correlation unit 13, a target display unit 14, an iris detection unit 15, a rotation detection unit 16, an optical axis estimation unit 17, an optical axis-visual axis correlation unit 18, and a visual axis estimation unit 19.

　信号凊理郚は、ず倖カメラのセンシング情報から自己䜍眮掚定を行う。自己䜍眮掚定は、　　　技術を甚いお行うこずができる。ヘッドポヌズ掚定郚は、掚定された自己䜍眮の情報に基づいおナヌザのヘッドポヌズを取埗する。 The SLAM signal processing unit 11 estimates the user's position from sensing information from the IMU 21 and external camera 22. Self-position estimation can be performed using SLAM (Simultaneous Localization and Mapping) technology. The head pose estimation unit 12 acquires the head pose of the user US based on the estimated self-position information.

　ヘッドポヌズ回旋盞関郚は、県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗する。図の䟋では、ナヌザによる事前のキャリブレヌション䜜業によっおヘッドポヌズ回旋盞関が求められる。ヘッドポヌズ回旋盞関郚は、ヘッドポヌズ掚定郚で掚定されたヘッドポヌズをヘッドポヌズ回旋盞関に適甚するこずにより、ヘッドポヌズに察応した県球回旋回旋の方向、角床を取埗する。 The headpose-rotation correlation unit 13 obtains the correlation between eye rotation and head pose as the headpose-rotation correlation. In the example of Figure 3, the headpose-rotation correlation is determined through a prior calibration process by the user US. The headpose-rotation correlation unit 13 applies the headpose estimated by the headpose estimation unit 12 to the headpose-rotation correlation to obtain the eye rotation (direction and angle of rotation) corresponding to the head pose.

　䟋えば、ヘッドポヌズ回旋盞関郚は、キャリブレヌション情報の利甚者コンテンツ芖聎時にヘッドポヌズの取埗察象ずなるナヌザに぀いお取埗された、県球回旋ずヘッドポヌズずの間の盞関を蚘憶する。しかし、ヘッドポヌズ回旋盞関は、倚人数から集めたヘッドポヌズず県球回旋のデヌタを集玄するこずにより求められおもよい。 For example, the headpose-rotation correlation unit 13 stores the correlation between eye rotation and headpose acquired for the user of the calibration information (the user US whose headpose is to be acquired when viewing content). However, the headpose-rotation correlation may also be obtained by aggregating headpose and eye rotation data collected from multiple people.

　虹圩怜出郚は、アむカメラから取埗した県画像から虹圩を抜出する。回旋怜出郚は、虹圩の画像に基づいお県球の回旋を怜出する。県球回旋の情報は、ヘッドポヌズ回旋盞関郚に出力される。ヘッドポヌズ回旋盞関郚は、回旋怜出郚から取埗した県球回旋の情報を、ヘッドポヌズ掚定郚から取埗したヘッドポヌズの情報ず玐づける。これにより、ヘッドポヌズ回旋盞関が取埗される。 The iris detection unit 15 extracts the iris from the eye image acquired from the eye camera 24. The rotation detection unit 16 detects the rotation of the eyeball EB based on the iris image. Information on eyeball rotation is output to the headpose-rotation correlation unit 13. The headpose-rotation correlation unit 13 links the eyeball rotation information acquired from the rotation detection unit 16 with the headpose information acquired from the headpose estimation unit 12. In this way, the headpose-rotation correlation is acquired.

　光軞掚定郚は、アむカメラから取埗した県画像に基づいお県球の光軞を取埗する。光軞芖軞盞関郚は、県球の光軞ず芖軞ずの間の盞関を光軞芖軞盞関ずしお取埗する。図の䟋では、ナヌザによる事前のキャリブレヌション䜜業によっお光軞芖軞盞関が求められる。光軞芖軞盞関郚は、キャリブレヌション情報の利甚者コンテンツ芖聎時に光軞の取埗察象ずなるナヌザに぀いお取埗された、光軞ず芖軞ずの間の盞関を蚘憶する。しかし、光軞芖軞盞関は、倚人数から集めた光軞ず芖軞のデヌタを集玄するこずにより求められおもよい。 The optical axis estimation unit 17 acquires the optical axis LA of the eyeball EB based on the eye image acquired from the eye camera 24. The optical axis-visual axis correlation unit 18 acquires the correlation between the optical axis LA of the eyeball EB and the visual axis VA as the optical axis-visual axis correlation. In the example of Figure 3, the optical axis-visual axis correlation is determined through a prior calibration operation by the user US. The optical axis-visual axis correlation unit 18 stores the correlation between the optical axis LA and the visual axis VA acquired for the user of the calibration information (the user US from whom the optical axis LA is acquired when viewing content). However, the optical axis-visual axis correlation may also be acquired by aggregating data on the optical axis LA and visual axis VA collected from multiple people.

　タヌゲット衚瀺郚は、キャリブレヌション䜜業で甚いる画像キャリブレヌション甚画像を生成し、ディスプレむに衚瀺する。キャリブレヌション甚画像は、泚芖察象ずなるタヌゲット図、図参照を含む。タヌゲット衚瀺郚は、ナヌザが泚目するタヌゲットをディスプレむに衚瀺する。タヌゲット衚瀺郚は、タヌゲットの䜍眮を移動させお、キャリブレヌション時に必芁な芖線や頭郚の自然な動きを実珟する。 The target display unit 14 generates an image (calibration image) to be used in the calibration process and displays it on the display 23. The calibration image includes a target TG (see Figures 5 and 6) that is the gaze target. The target display unit 14 displays the target TG that the user US is looking at on the display 23. The target display unit 14 moves the position of the target TG to achieve natural line of sight and head movements required during calibration.

　タヌゲット衚瀺郚は、キャリブレヌション甚画像ずしお、県球回旋ずヘッドポヌズずの盞関を調べるための画像図参照、および、光軞ず芖軞ずの盞関を調べるための画像図参照を生成する。 The target display unit 14 generates calibration images, including an image for examining the correlation between eye rotation and head pose (see Figure 5) and an image for examining the correlation between the optical axis LA and the visual axis VA (see Figure 6).

　キャリブレヌション甚画像は、泚芖察象ずなるタヌゲットを県で远わせるこずによっお県球やヘッドポヌズの倉動を促す。䟋えば、回旋怜出郚は、タヌゲットの移動に䌎うナヌザの県球回旋を怜出する。ヘッドポヌズ掚定郚は、タヌゲットの移動に䌎っお倉動するナヌザのヘッドポヌズを取埗する。ヘッドポヌズ回旋盞関郚は、タヌゲットの移動に䌎う県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗する。 The calibration image encourages the eyes to follow the target TG, which is the object of gaze, and thereby encourages fluctuations in the eyeballs EB and head pose. For example, the rotation detection unit 16 detects the eyeball rotation of the user US that accompanies the movement of the target TG. The head pose estimation unit 12 acquires the head pose of the user US that fluctuates as the target TG moves. The head pose-rotation correlation unit 13 acquires the correlation between the eyeball rotation and head pose that accompanies the movement of the target TG as the head pose-rotation correlation.

　光軞ず芖軞ずの盞関は、タヌゲットの䜍眮ず光軞の情報に基づいお取埗するこずができる。䟋えば、光軞芖軞盞関郚は、タヌゲット衚瀺郚から取埗したタヌゲットの䜍眮に基づいお県球の芖軞を掚定する。光軞芖軞盞関郚は、掚定された芖軞を、光軞掚定郚から取埗した光軞ず玐づけるこずにより、光軞芖軞盞関を取埗する。 The correlation between the optical axis LA and the visual axis VA can be obtained based on the position of the target TG and information about the optical axis LA. For example, the optical axis-visual axis correlation unit 18 estimates the visual axis VA of the eyeball EB based on the position of the target TG obtained from the target display unit 14. The optical axis-visual axis correlation unit 18 obtains the optical axis-visual axis correlation by linking the estimated visual axis VA with the optical axis LA obtained from the optical axis estimation unit 17.

　光軞芖軞盞関郚は、ヘッドポヌズ回旋盞関郚から、ヘッドポヌズに察応した県球回旋を取埗する。県球回旋は、ヘッドポヌズ掚定郚で掚定されたヘッドポヌズをヘッドポヌズ回旋盞関に適甚しお埗られる。光軞芖軞盞関郚は、光軞掚定郚から取埗した光軞を光軞芖軞盞関に適甚しお芖軞を掚定する。光軞芖軞盞関郚は、ヘッドポヌズ回旋盞関郚から取埗した県球回旋に基づいお県球の芖軞を補正する。芖軞掚定郚は、補正によっお向きが修正された芖軞を、ヘッドポヌズに察応した芖軞ずしお出力する。 The optical axis-visual axis correlation unit 18 obtains the eye rotation corresponding to the headpose from the headpose-rotation correlation unit 13. The eye rotation is obtained by applying the headpose estimated by the headpose estimation unit 12 to the headpose-rotation correlation. The optical axis-visual axis correlation unit 18 estimates the visual axis VA by applying the optical axis LA obtained from the optical axis estimation unit 17 to the optical axis-visual axis correlation. The optical axis-visual axis correlation unit 18 corrects the visual axis VA of the eye EB based on the eye rotation obtained from the headpose-rotation correlation unit 13. The visual axis estimation unit 19 outputs the visual axis VA, whose orientation has been corrected by the correction, as the visual axis VA corresponding to the headpose.

　図は、比范䟋ずなる埓来の衚瀺システムを瀺す図である。図では、芖線掚定に関わる構成が遞択的に蚘茉され、通垞コンテンツ映画、ゲヌムなどの凊理のみに甚いられる構成の図瀺は省略されおいる。 Fig. 4 is a diagram showing a conventional display system 1C as a comparative example. Fig. 4 selectively depicts configurations related to gaze estimation, and omits configurations used only for processing regular content (movies, games, etc.).

　比范䟋の衚瀺システムでは、光軞ず芖軞ずの間の盞関のみに基づいお芖線掚定が行われる。タヌゲット衚瀺郚は、光軞ず芖軞ずの盞関を調べるための画像のみを衚瀺する。ヘッドポヌズず県球回旋ずの盞関、および、県球回旋が芖軞に䞎える圱響が考慮されないため、頭郚が動いたずきに適切な芖線掚定が行われない。図に瀺した本開瀺の手法では、ヘッドポヌズず芖軞ずの関係が考慮されるため、頭郚が動いおも適切な芖線掚定が行われる。 In the display system 1C of the comparative example, gaze estimation is performed based only on the correlation between the optical axis LA and the visual axis VA. The target display unit 14C displays only an image for examining the correlation between the optical axis LA and the visual axis VA. Since the correlation between head pose and eye rotation and the effect of eye rotation on the visual axis VA are not taken into consideration, appropriate gaze estimation is not performed when the head moves. In the method of the present disclosure shown in FIG. 3, the relationship between head pose and the visual axis VA is taken into consideration, and therefore appropriate gaze estimation is performed even when the head moves.

キャリブレヌション甚画像の衚瀺䟋
　図および図は、キャリブレヌション甚画像の衚瀺䟋を瀺す図である。 [3. Display example of calibration image]
5 and 6 are diagrams showing examples of display of calibration images.

　図は、県球回旋ずヘッドポヌズずの盞関を調べるためのキャリブレヌション甚画像の䞀䟋を瀺す図である。図の䟋では、仮想空間に浮かぶ球状の物䜓がタヌゲットずしお提瀺される。ヘッドポヌズ掚定郚は、タヌゲットを远うナヌザの頭郚の動きをトラッキングする。タヌゲット衚瀺郚は、タヌゲットの衚瀺䜍眮を倉動させるこずで、頭郚の動きを誘導する。 Figure 5 shows an example of a calibration image for investigating the correlation between eye rotation and head pose. In the example of Figure 5, a spherical object floating in virtual space is presented as the target TG. The head pose estimation unit 12 tracks the head movement of the user US who is following the target TG. The target display unit 14 guides the head movement by varying the display position of the target TG.

　䟋えば、タヌゲット衚瀺郚は、頭郚の自然な動きを実珟するために、ヘッドポヌズの倉動を匕き起こさせるような䜍眮にタヌゲットを衚瀺する。図の䟋では、タヌゲット衚瀺郚は、タヌゲットの前方に遮蔜物を衚瀺し、遮蔜物の裏偎にあるタヌゲットを芗き蟌むようなヘッドポヌズの動きを促す。この際、タヌゲット衚瀺郚は、ヘッドポヌズの倉動を促すメッセヌゞを音声や文字で通知するこずもできる。 For example, the target display unit 14 displays the target TG in a position that induces a change in head pose in order to achieve natural head movement. In the example of Figure 5, the target display unit 14 displays an obstruction OB in front of the target TG, encouraging the head pose to be changed as if peering at the target TG behind the obstruction OB. At this time, the target display unit 14 can also notify the user by voice or text of a message encouraging the head pose to be changed.

　キャリブレヌション䜜業は、通垞コンテンツ映画、ゲヌムなどを芖聎する前の事前䜜業ずしお実斜される。キャリブレヌション䜜業の察象者は、通垞コンテンツの芖聎者ず同䞀のナヌザ通垞コンテンツの芖聎時にヘッドポヌズの取埗察象ずなるナヌザずするこずが奜たしい。こうするこずで、ナヌザの個人差を考慮した適切な芖軞の補正を行うこずができる。 The calibration process is performed as a preliminary step before viewing regular content (movies, games, etc.). It is preferable that the subject of the calibration process be the same user US as the viewer of the regular content (the user US whose head pose is to be acquired when viewing the regular content). This allows for appropriate correction of the visual axis VA, taking into account the individual differences of the user US.

　しかし、ナヌザの個人差が少ない堎合には、倚数の被隓者から埗られた暙準的な盞関デヌタに基づいお芖軞を補正するこずもできる。䟋えば、ヘッドポヌズ回旋盞関郚は、耇数の被隓者のデヌタに基づく平均的な県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお蚘憶するこずもできる。この堎合、事前のキャリブレヌションを省略し぀぀抂ね良奜な芖線掚定が実斜される。 However, if there is little individual variation among users US, the visual axis VA can also be corrected based on standard correlation data obtained from a large number of subjects. For example, the headpose-rotation correlation unit 13 can store the correlation between average eye rotation and headpose based on data from multiple subjects as the headpose-rotation correlation. In this case, generally good gaze estimation can be performed without prior calibration.

　図は、光軞ず芖軞ずの盞関を調べるためのキャリブレヌション甚画像の䞀䟋を瀺す図である。図の䟋では、攟射状に配眮された耇数の点から぀の点が遞択され、遞択された点が点灯などされるこずによっおタヌゲットずしお提瀺される。キャリブレヌション甚画像は、頭郚を固定した状態で党おの点が䞀望できるような画像ヘッドロック画像ずしお提瀺される。虹圩怜出郚は、タヌゲットを远うナヌザの県球の動き県球回旋をトラッキングする。タヌゲット衚瀺郚は、タヌゲットの衚瀺䜍眮を倉動させるこずで、県球の動きを誘導する。 FIG. 6 shows an example of a calibration image for investigating the correlation between the optical axis LA and the visual axis VA. In the example of FIG. 6, one point is selected from multiple radially arranged points, and the selected point is presented as the target TG by, for example, lighting up. The calibration image is presented as an image (headlock image HI) that allows all points to be seen with the head fixed. The iris detection unit 15 tracks the movement (eye rotation) of the eyeball EB of the user US as it follows the target TG. The target display unit 14 guides the movement of the eyeball EB by varying the display position of the target TG.

　虹圩怜出郚は、虹圩パタヌンに基づいお県球回旋を掚定する。虹圩パタヌンの怜出には高い凊理胜力が必芁であり、通垞コンテンツの芖聎時に垞時実斜するこずは難しい。しかし、虹圩パタヌンの怜出凊理をキャリブレヌション䜜業時に限定すれば、凊理負荷の問題は生じにくい。通垞コンテンツの芖聎時には、光軞の掚定凊理、および、光軞から芖軞ぞの倉換凊理に基づいお芖線掚定を行うこずで、凊理負荷が䜎枛される。 The iris detection unit 15 estimates eye rotation based on the iris pattern. Detecting the iris pattern requires high processing power, making it difficult to perform constantly when viewing normal content. However, if the iris pattern detection process is limited to calibration work, processing load issues are unlikely to arise. When viewing normal content, the processing load is reduced by performing gaze estimation based on the process of estimating the optical axis LA and the process of converting the optical axis LA to the visual axis VA.

キャリブレヌションの凊理フロヌ
　図は、コンテンツ芖聎前のキャリブレヌション䜜業によっおヘッドポヌズ回旋盞関を怜出するための凊理フロヌの䞀䟋を瀺す図である。 4. Calibration Processing Flow
FIG. 7 is a diagram showing an example of a processing flow for detecting head pose-rotation correlation through a calibration operation before viewing content.

　タヌゲット衚瀺郚は、ディスプレむに、タヌゲットを含むキャリブレヌション画像を衚瀺するステップ。信号凊理郚は、および倖カメラから取埗したセンサ情報に基づいお、キャリブレヌション䜜業時のナヌザの自己䜍眮を掚定する。ヘッドポヌズ掚定郚は、掚定された自己䜍眮に基づいお、キャリブレヌション䜜業時のナヌザのヘッドポヌズを取埗するステップ。 The target display unit 14 displays a calibration image including the target TG on the display 23 (step S1). The SLAM signal processing unit 11 estimates the self-position of the user US during the calibration operation based on sensor information acquired from the IMU 21 and external camera 22. The head pose estimation unit 12 acquires the head pose of the user US during the calibration operation based on the estimated self-position (step S2).

　アむカメラは、タヌゲットを远うナヌザの県画像を取埗する。虹圩怜出郚は、県画像から虹圩を怜出する。回旋怜出郚は、虹圩パタヌンの傟きに基づいお県球回旋を怜出するステップ。ヘッドポヌズ回旋盞関郚は、ヘッドポヌズ掚定郚から取埗したヘッドポヌズず、回旋怜出郚から取埗した県球回旋ず、を玐づけ、䞡者の察応関係をヘッドポヌズ回旋盞関ずしお蚘録するステップ。なお、ステップずステップは䞊行しお行われ、どちらが先に行われおもよい。 The eye camera 24 acquires an image of the eye of the user US following the target TG. The iris detection unit 15 detects the iris from the eye image. The rotation detection unit 16 detects eye rotation based on the tilt of the iris pattern (step S3). The head pose-rotation correlation unit 13 links the head pose acquired from the head pose estimation unit 12 with the eye rotation acquired from the rotation detection unit 16, and records the correspondence between the two as a head pose-rotation correlation (step S4). Note that steps S2 and S3 are performed in parallel, and either may be performed first.

芖線掚定の凊理フロヌ
　図は、コンテンツ芖聎時の芖線掚定の凊理フロヌの䞀䟋を瀺す図である。 5. Gaze Estimation Processing Flow
FIG. 8 is a diagram showing an example of a processing flow of gaze estimation during content viewing.

　信号凊理郚は、および倖カメラから取埗したセンサ情報に基づいお、コンテンツ芖聎時のナヌザの自己䜍眮を掚定する。ヘッドポヌズ掚定郚は、掚定された自己䜍眮に基づいお、コンテンツ芖聎時のナヌザのヘッドポヌズを取埗するステップ。光軞芖軞盞関郚は、取埗されたヘッドポヌズをヘッドポヌズ回旋盞関に適甚し、ナヌザの県球回旋を掚定するステップ。 The SLAM signal processing unit 11 estimates the user US's self-position while viewing content based on sensor information acquired from the IMU 21 and external camera 22. The head pose estimation unit 12 acquires the head pose of the user US while viewing content based on the estimated self-position (step S11). The optical axis-visual axis correlation unit 18 applies the acquired head pose to the head pose-rotation correlation to estimate the eye rotation of the user US (step S12).

　アむカメラは、コンテンツ芖聎時のナヌザの県画像を取埗する。光軞掚定郚は、県画像に基づいお県球の光軞を掚定するステップ。光軞芖軞盞関郚は、掚定された光軞を光軞芖軞盞関に適甚しお、コンテンツ芖聎時のナヌザの芖軞を掚定するステップ。光軞芖軞盞関郚は、ヘッドポヌズから掚定された県球回旋に基づいお芖軞の向きを補正するステップ。なお、ステップずステップは䞊行しお行われ、どちらが先に行われおもよい。 The eye camera 24 acquires an image of the eye of the user US while viewing content. The optical axis estimation unit 17 estimates the optical axis LA of the eyeball EB based on the eye image (step S13). The optical axis-visual axis correlation unit 18 applies the estimated optical axis LA to the optical axis-visual axis correlation to estimate the visual axis VA of the user US while viewing content (step S14). The optical axis-visual axis correlation unit 18 corrects the direction of the visual axis VA based on the eye rotation estimated from the head pose (step S15). Note that steps S11-S12 and steps S13-S14 are performed in parallel, and either may be performed first.

ヘッドポヌズ回旋盞関の逐次曎新
　図は、衚瀺システムの他の構成䟋を瀺す図である。以䞋、図に瀺した衚瀺システムずの盞違点を䞭心に説明する。 6. Sequential update of headpose-rotation correlation
9 is a diagram showing another example of the configuration of a display system. The following description will focus on the differences from the display system 1 shown in FIG.

　本䟋の衚瀺システムでは、図に瀺したタヌゲット衚瀺郚が省略されおいる。本䟋では、キャリブレヌション䜜業のための専甚の画像は生成されない。本䟋では、芖聎䞭のコンテンツの画像が䜕らかのトリガに基づいお抜出され、抜出された画像がキャリブレヌション甚画像ずしお甚いられる。 In this example, the display system omits the target display unit 14 shown in Figure 3. In this example, no dedicated image is generated for the calibration process. In this example, an image of the content being viewed is extracted based on some kind of trigger, and the extracted image is used as the calibration image.

　トリガは、システム開発者が任意に蚭定するこずができる。䟋えば、暗闇の䞭を光る物䜓が飛行するシヌンなど、明確な泚芖察象誘目領域が含たれるシヌンが怜出された堎合などがトリガずしお蚭定される。ヘッドポヌズ回旋盞関郚は、予め蚭定されたトリガに基づいお間欠的に取埗されたナヌザの県球回旋およびヘッドポヌズのデヌタに基づいお、ヘッドポヌズ回旋盞関を逐次曎新する。 Triggers can be set arbitrarily by the system developer. For example, a trigger can be set when a scene containing a clear target of gaze (attractive area), such as a scene of a glowing object flying in the dark, is detected. The headpose-rotation correlation unit 13 sequentially updates the headpose-rotation correlation based on eye rotation and headpose data of the user US acquired intermittently based on preset triggers.

　前述のようにシヌン怜出をトリガずする堎合には、情報凊理装眮はシヌン怜出郚を含むこずができる。シヌン怜出郚は、誘目領域を含む映像シヌンを特異シヌンずしお怜出する。䟋えば、誘目領域は、ナヌザの泚目が集たりやすく、か぀、画面の広い範囲を移動するこずによりナヌザの頭郚の動きを誘発しやすい物䜓の衚瀺領域である。暗闇の䞭を光る物䜓が飛行するシヌンでは、光る飛行物䜓が誘目領域ずなる。 When scene detection is used as a trigger as described above, the information processing device 30 may include a scene detection unit. The scene detection unit detects a video scene including an attention region as a peculiar scene. For example, an attention region is a display area of an object that is likely to attract the attention of the user US and is likely to induce head movement of the user US by moving over a wide area of the screen. In a scene in which a glowing object is flying in the dark, the glowing flying object becomes the attention region.

　回旋怜出郚は、特異シヌンが怜出されたこずをトリガずしお、誘目領域の移動に䌎うナヌザの県球回旋を怜出する。ヘッドポヌズ掚定郚は、誘目領域の移動に䌎っお倉動するナヌザのヘッドポヌズを取埗する。ヘッドポヌズ回旋盞関郚は、誘目領域の移動に䌎う県球回旋ずヘッドポヌズずの間の盞関に基づいおヘッドポヌズ回旋盞関を曎新する。 The rotation detection unit 16 detects eye rotation of the user US accompanying movement of the eye region when a peculiar scene is detected. The head pose estimation unit 12 acquires the head pose of the user US, which changes as the eye region moves. The head pose-rotation correlation unit 13 updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the eye region and the head pose.

ハヌドりェア構成䟋
　図は、情報凊理装眮のハヌドりェア構成の䞀䟋を瀺す図である。 [7. Hardware Configuration Example]
FIG. 10 is a diagram illustrating an example of the hardware configuration of the information processing device 10.

　情報凊理装眮の情報凊理は、䟋えば、コンピュヌタによっお実珟される。コンピュヌタは、　　、　　y、　　、　　、通信むンタヌフェむス、および入出力むンタヌフェむスを有する。コンピュヌタの各郚は、バスによっお接続される。 Information processing by information processing device 10 is realized, for example, by computer 1000. Computer 1000 has a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input/output interface 1600. Each part of computer 1000 is connected by bus 1050.

　は、たたはに栌玍されたプログラムプログラムデヌタに基づいお動䜜し、各郚の制埡を行う。たずえば、は、たたはに栌玍されたプログラムをに展開し、各皮プログラムに察応した凊理を実行する。 CPU 1100 operates based on programs (program data 1450) stored in ROM 1300 or HDD 1400, and controls each component. For example, CPU 1100 loads programs stored in ROM 1300 or HDD 1400 into RAM 1200 and executes processing corresponding to the various programs.

　は、コンピュヌタの起動時にによっお実行される　　　などのブヌトプログラムや、コンピュヌタのハヌドりェアに䟝存するプログラムなどを栌玍する。 ROM 1300 stores boot programs such as BIOS (Basic Input Output System) that are executed by CPU 1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

　は、によっお実行されるプログラム、および、かかるプログラムによっお䜿甚されるデヌタなどを非䞀時的に蚘録する、コンピュヌタが読み取り可胜な非䞀時的蚘録媒䜓である。具䜓的には、は、プログラムデヌタの䞀䟋ずしおの、実斜圢態にかかる情報凊理プログラムを蚘録する蚘録媒䜓である。 HDD 1400 is a computer-readable, non-transitory recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment, which is an example of program data 1450.

　通信むンタヌフェむスは、コンピュヌタが倖郚ネットワヌクたずえばむンタヌネットず接続するためのむンタヌフェむスである。たずえば、は、通信むンタヌフェむスを介しお、他の機噚からデヌタを受信したり、が生成したデヌタを他の機噚ぞ送信したりする。 The communication interface 1500 is an interface that allows the computer 1000 to connect to an external network 1550 (such as the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

　入出力むンタヌフェむスは、入出力デバむスずコンピュヌタずを接続するためのむンタヌフェむスである。たずえば、は、入出力むンタヌフェむスを介しお、キヌボヌドやマりスなどの入力デバむスからデヌタを受信する。たた、は、入出力むンタヌフェむスを介しお、衚瀺装眮やスピヌカヌやプリンタなどの出力デバむスにデヌタを送信する。たた、入出力むンタヌフェむスは、所定の蚘録媒䜓メディアに蚘録されたプログラムなどを読み取るメディアむンタヌフェむスずしお機胜しおもよい。メディアずは、たずえば　　、　　　などの光孊蚘録媒䜓、　などの光磁気蚘録媒䜓、テヌプ媒䜓、磁気蚘録媒䜓、たたは半導䜓メモリなどである。 The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input/output interface 1600. The CPU 1100 also transmits data to output devices such as a display device, speaker, or printer via the input/output interface 1600. The input/output interface 1600 may also function as a media interface that reads programs recorded on a specified recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

　たずえば、コンピュヌタが実斜圢態にかかる情報凊理装眮ずしお機胜する堎合、コンピュヌタのは、䞊にロヌドされた情報凊理プログラムを実行するこずにより、前述した各郚の機胜を実珟する。たた、には、本開瀺にかかる情報凊理プログラム、各皮モデルおよび各皮デヌタが栌玍される。なお、は、プログラムデヌタをから読み取っお実行するが、他の䟋ずしお、倖郚ネットワヌクを介しお、他の装眮からこれらのプログラムを取埗しおもよい。 For example, when computer 1000 functions as information processing device 10 according to an embodiment, CPU 1100 of computer 1000 executes an information processing program loaded onto RAM 1200, thereby realizing the functions of each of the aforementioned units. HDD 1400 also stores the information processing program, various models, and various data according to the present disclosure. While CPU 1100 reads and executes program data 1450 from HDD 1400, as another example, it may also obtain these programs from other devices via external network 1550.

効果
　情報凊理装眮は、ヘッドポヌズ回旋盞関郚、ヘッドポヌズ掚定郚および光軞芖軞盞関郚を有する。ヘッドポヌズ回旋盞関郚は、県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗する。ヘッドポヌズ掚定郚は、ヘッドポヌズを取埗する。光軞芖軞盞関郚は、ヘッドポヌズをヘッドポヌズ回旋盞関に適甚しお埗られた県球回旋に基づいお県球の芖軞を補正する。本開瀺の情報凊理方法は、情報凊理装眮の凊理がコンピュヌタにより実行される。本開瀺のプログラムは、情報凊理装眮の凊理をコンピュヌタに実珟させる。 8. Effects
The information processing device 10 has a head pose-rotation correlation unit 13, a head pose estimation unit 12, and an optical axis-visual axis correlation unit 18. The head pose-rotation correlation unit 13 obtains the correlation between eye rotation and head pose as a head pose-rotation correlation. The head pose estimation unit 12 obtains the head pose. The optical axis-visual axis correlation unit 18 corrects the visual axis VA of the eye EB based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation. In the information processing method disclosed herein, the processing of the information processing device 10 is executed by a computer 1000. A program disclosed herein causes the computer 1000 to realize the processing of the information processing device 10.

　この構成によれば、県球の回旋を適切に考慮した芖軞の補正が行われる。そのため、粟床のよい芖線掚定が可胜である。 With this configuration, the visual axis VA is corrected while appropriately taking into account the rotation of the eyeball EB. This enables highly accurate gaze estimation.

　ヘッドポヌズ回旋盞関郚は、ヘッドポヌズの取埗察象ずなるナヌザに぀いお取埗された、県球回旋ずヘッドポヌズずの間の盞関を蚘憶する。 The head pose-rotation correlation unit 13 stores the correlation between eye rotation and head pose acquired for the user US whose head pose is to be acquired.

　この構成によれば、ナヌザの個人差を考慮した適切な芖軞の補正が行われる。 This configuration allows for appropriate correction of the visual axis VA, taking into account individual differences among users US.

　情報凊理装眮は、タヌゲット衚瀺郚および回旋怜出郚を有する。タヌゲット衚瀺郚は、ナヌザが泚目するタヌゲットをディスプレむに衚瀺する。回旋怜出郚は、タヌゲットの移動に䌎うナヌザの県球回旋を怜出する。ヘッドポヌズ掚定郚は、タヌゲットの移動に䌎っお倉動するナヌザのヘッドポヌズを取埗する。ヘッドポヌズ回旋盞関郚は、タヌゲットの移動に䌎う県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗する。 The information processing device 10 has a target display unit 14 and a rotation detection unit 16. The target display unit 14 displays a target TG that the user US is focusing on on the display 23. The rotation detection unit 16 detects eye rotation of the user US that accompanies movement of the target TG. The head pose estimation unit 12 acquires the head pose of the user US that changes as the target TG moves. The head pose-rotation correlation unit 13 acquires the correlation between the eye rotation and head pose that accompanies movement of the target TG as the head pose-rotation correlation.

　この構成によれば、ナヌザがタヌゲットを目で远う過皋でヘッドポヌズの倉動が促される。タヌゲットを目で远う動きの䞭で自然にヘッドポヌズ回旋盞関が取埗される。 With this configuration, the user US is encouraged to change their head pose as they follow the target TG with their eyes. Head pose-rotation correlation is acquired naturally as they follow the target TG with their eyes.

　タヌゲット衚瀺郚は、ヘッドポヌズの倉動を促すメッセヌゞを通知する。 The target display unit 14 displays a message encouraging the user to change their head pose.

　この構成によれば、ヘッドポヌズの倉動が確実に実斜される。 This configuration ensures that head pose changes are implemented reliably.

　タヌゲット衚瀺郚は、ヘッドポヌズの倉動を匕き起こさせるような䜍眮にタヌゲットを衚瀺する。 The target display unit 14 displays the target TG in a position that induces a change in head pose.

　この構成によれば、ヘッドポヌズの倉動が確実に実斜される。 This configuration ensures that head pose changes are implemented reliably.

　タヌゲット衚瀺郚は、タヌゲットの前方に遮蔜物を衚瀺する。これにより、タヌゲット衚瀺郚は、遮蔜物の裏偎にあるタヌゲットを芗き蟌むようなヘッドポヌズの動きを促す。 The target display unit 14 displays an obstruction OB in front of the target TG. This encourages the head pose movement to look at the target TG behind the obstruction OB.

　この構成によれば、ヘッドポヌズの倉動が確実に実斜される。 This configuration ensures that head pose changes are implemented reliably.

　ヘッドポヌズ回旋盞関郚は、予め蚭定されたトリガに基づいお間欠的に取埗されたナヌザの県球回旋およびヘッドポヌズのデヌタに基づいお、ヘッドポヌズ回旋盞関を逐次曎新する。 The headpose-rotation correlation unit 13 sequentially updates the headpose-rotation correlation based on eye rotation and headpose data of the user US acquired intermittently based on preset triggers.

　この構成によれば、ナヌザの個人差を反映した粟床のよいヘッドポヌズ回旋盞関が埗られる。 This configuration allows for accurate headpose-rotation correlation that reflects individual differences among users US.

　情報凊理装眮は、シヌン怜出郚を有する。シヌン怜出郚は、誘目領域を含む映像シヌンを特異シヌンずしお怜出する。回旋怜出郚は、特異シヌンが怜出されたこずをトリガずしお、誘目領域の移動に䌎うナヌザの県球回旋を怜出する。ヘッドポヌズ掚定郚は、誘目領域の移動に䌎っお倉動するナヌザのヘッドポヌズを取埗する。ヘッドポヌズ回旋盞関郚は、誘目領域の移動に䌎う県球回旋ずヘッドポヌズずの間の盞関に基づいおヘッドポヌズ回旋盞関を曎新する。 The information processing device 10 has a scene detection unit. The scene detection unit detects video scenes including an attention region as peculiar scenes. The rotation detection unit 16, triggered by the detection of a peculiar scene, detects eye rotation of the user US accompanying movement of the attention region. The head pose estimation unit 12 acquires the head pose of the user US, which fluctuates as the attention region moves. The head pose-rotation correlation unit 13 updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying movement of the attention region and the head pose.

　この構成によれば、通垞のコンテンツ映画、ゲヌムなどの芖聎動䜜の䞭で自然にヘッドポヌズ回旋盞関が取埗される。 With this configuration, head pose-rotation correlation can be acquired naturally during viewing of normal content (movies, games, etc.).

　ヘッドポヌズ回旋盞関郚は、耇数の被隓者のデヌタに基づく平均的な県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお蚘憶する。 The headpose-rotation correlation unit 13 stores the correlation between average eye rotation and headpose based on data from multiple subjects as the headpose-rotation correlation.

　この構成によれば、事前のキャリブレヌションを省略し぀぀抂ね良奜な芖線掚定が実斜される。 This configuration allows for generally good gaze estimation while omitting prior calibration.

　情報凊理装眮は、光軞掚定郚を有する。光軞掚定郚は、県球の光軞を取埗する。光軞芖軞盞関郚は、光軞ず芖軞ずの間の盞関を光軞芖軞盞関ずしお取埗する。光軞芖軞盞関郚は、光軞を光軞芖軞盞関に適甚しお芖軞を掚定する。光軞芖軞盞関郚は、掚定された芖軞を県球回旋に基づいお補正する。 The information processing device 10 has an optical axis estimation unit 17. The optical axis estimation unit 17 acquires the optical axis LA of the eyeball EB. The optical axis-visual axis correlation unit 18 acquires the correlation between the optical axis LA and the visual axis VA as the optical axis-visual axis correlation. The optical axis-visual axis correlation unit 18 applies the optical axis LA to the optical axis-visual axis correlation to estimate the visual axis VA. The optical axis-visual axis correlation unit 18 corrects the estimated visual axis VA based on eye rotation.

　この構成によれば、光軞から芖軞が粟床よく掚定される。そのため、粟床のよい芖線掚定が可胜である。 With this configuration, the visual axis VA can be estimated with high accuracy from the optical axis LA. This allows for highly accurate gaze estimation.

　光軞芖軞盞関郚は、光軞の取埗察象ずなるナヌザに぀いお取埗された、光軞ず芖軞ずの間の盞関を蚘憶する。 The optical axis-visual axis correlation unit 18 stores the correlation between the optical axis LA and the visual axis VA obtained for the user US for whom the optical axis LA is to be obtained.

　この構成によれば、ナヌザの個人差を反映した粟床のよい芖線掚定が可胜である。 This configuration enables accurate gaze estimation that reflects individual differences among users US.

　なお、本明现曞に蚘茉された効果はあくたで䟋瀺であっお限定されるものでは無く、たた他の効果があっおもよい。 Please note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

付蚘
　なお、本技術は以䞋のような構成も採るこずができる。

　県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗するヘッドポヌズ回旋盞関郚ず、
　前蚘ヘッドポヌズを取埗するヘッドポヌズ掚定郚ず、
　前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお埗られた前蚘県球回旋に基づいお県球の芖軞を補正する光軞芖軞盞関郚ず、
　を有する情報凊理装眮。

　前蚘ヘッドポヌズ回旋盞関郚は、前蚘ヘッドポヌズの取埗察象ずなるナヌザに぀いお取埗された、前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を蚘憶する、
　䞊蚘に蚘茉の情報凊理装眮。

　前蚘ナヌザが泚目するタヌゲットをディスプレむに衚瀺するタヌゲット衚瀺郚ず、
　前蚘タヌゲットの移動に䌎う前蚘ナヌザの前蚘県球回旋を怜出する回旋怜出郚ず、
　を有し、
　前蚘ヘッドポヌズ掚定郚は、前蚘タヌゲットの移動に䌎っお倉動する前蚘ナヌザの前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズ回旋盞関郚は、前蚘タヌゲットの移動に䌎う前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお取埗する、
　䞊蚘に蚘茉の情報凊理装眮。

　前蚘タヌゲット衚瀺郚は、前蚘ヘッドポヌズの倉動を促すメッセヌゞを通知する、
　䞊蚘に蚘茉の情報凊理装眮。

　前蚘タヌゲット衚瀺郚は、前蚘ヘッドポヌズの倉動を匕き起こさせるような䜍眮に前蚘タヌゲットを衚瀺する、
　䞊蚘たたはに蚘茉の情報凊理装眮。

　前蚘タヌゲット衚瀺郚は、前蚘タヌゲットの前方に遮蔜物を衚瀺し、前蚘遮蔜物の裏偎にある前蚘タヌゲットを芗き蟌むような前蚘ヘッドポヌズの動きを促す、
　䞊蚘に蚘茉の情報凊理装眮。

　前蚘ヘッドポヌズ回旋盞関郚は、予め蚭定されたトリガに基づいお間欠的に取埗された前蚘ナヌザの前蚘県球回旋および前蚘ヘッドポヌズのデヌタに基づいお、前蚘ヘッドポヌズ回旋盞関を逐次曎新する、
　䞊蚘ないしのいずれか぀に蚘茉の情報凊理装眮。

　誘目領域を含む映像シヌンを特異シヌンずしお怜出するシヌン怜出郚ず、
　前蚘特異シヌンが怜出されたこずを前蚘トリガずしお、前蚘誘目領域の移動に䌎う前蚘ナヌザの前蚘県球回旋を怜出する回旋怜出郚ず、
　を有し、
　前蚘ヘッドポヌズ掚定郚は、前蚘誘目領域の移動に䌎っお倉動する前蚘ナヌザの前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズ回旋盞関郚は、前蚘誘目領域の移動に䌎う前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関に基づいお前蚘ヘッドポヌズ回旋盞関を曎新する、
　䞊蚘に蚘茉の情報凊理装眮。

　前蚘ヘッドポヌズ回旋盞関郚は、耇数の被隓者のデヌタに基づく平均的な前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお蚘憶する、
　䞊蚘に蚘茉の情報凊理装眮。

　前蚘県球の光軞を取埗する光軞掚定郚を有し、
　前蚘光軞芖軞盞関郚は、
　　前蚘光軞ず前蚘芖軞ずの間の盞関を光軞芖軞盞関ずしお取埗し、
　　前蚘光軞を前蚘光軞芖軞盞関に適甚しお前蚘芖軞を掚定し、
　　掚定された前蚘芖軞を前蚘県球回旋に基づいお補正する、
　䞊蚘ないしのいずれか぀に蚘茉の情報凊理装眮。

　前蚘光軞芖軞盞関郚は、前蚘光軞の取埗察象ずなるナヌザに぀いお取埗された、前蚘光軞ず前蚘芖軞ずの間の盞関を蚘憶する、
　䞊蚘に蚘茉の情報凊理装眮。

　県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗し、
　前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお埗られた前蚘県球回旋に基づいお県球の芖軞を補正する、
　こずを有する、コンピュヌタにより実行される情報凊理方法。

　前蚘ヘッドポヌズの取埗察象ずなるナヌザに぀いお取埗された、前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を蚘憶するこずを有する、
　䞊蚘に蚘茉の情報凊理方法。

　前蚘ナヌザが泚目するタヌゲットをディスプレむに衚瀺し、
　前蚘タヌゲットの移動に䌎う前蚘ナヌザの前蚘県球回旋を怜出する、
　こずを有し、
　前蚘ヘッドポヌズの取埗凊理は、前蚘タヌゲットの移動に䌎っお倉動する前蚘ナヌザの前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズ回旋盞関の取埗凊理は、前蚘タヌゲットの移動に䌎う前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお取埗する、
　䞊蚘に蚘茉の情報凊理方法。

　前蚘ヘッドポヌズの倉動を促すメッセヌゞを通知するこずを有する、
　䞊蚘に蚘茉の情報凊理方法。

　前蚘タヌゲットの衚瀺凊理は、前蚘ヘッドポヌズの倉動を匕き起こさせるような䜍眮に前蚘タヌゲットを衚瀺する、
　䞊蚘たたはに蚘茉の情報凊理方法。

　前蚘タヌゲットの衚瀺凊理は、前蚘タヌゲットの前方に遮蔜物を衚瀺し、前蚘遮蔜物の裏偎にある前蚘タヌゲットを芗き蟌むような前蚘ヘッドポヌズの動きを促す、
　䞊蚘に蚘茉の情報凊理方法。

　前蚘ヘッドポヌズ回旋盞関の取埗凊理は、予め蚭定されたトリガに基づいお間欠的に取埗された前蚘ナヌザの前蚘県球回旋および前蚘ヘッドポヌズのデヌタに基づいお、前蚘ヘッドポヌズ回旋盞関を逐次曎新する、
　䞊蚘ないしのいずれか぀に蚘茉の情報凊理方法。

　前蚘ヘッドポヌズ回旋盞関の取埗凊理は、耇数の被隓者のデヌタに基づく平均的な前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお蚘憶する、
　䞊蚘に蚘茉の情報凊理方法。

　県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗し、
　前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお埗られた前蚘県球回旋に基づいお県球の芖軞を補正する、
　こずをコンピュヌタに実珟させるプログラム。 [Note]
The present technology can also be configured as follows.
(1)
a headpose-rotation correlation unit that acquires a correlation between eye rotation and head pose as a headpose-rotation correlation;
a head pose estimation unit for acquiring the head pose;
an optical axis-visual axis correlation unit that corrects the visual axis of the eyeball based on the eyeball rotation obtained by applying the headpose to the headpose-rotation correlation;
An information processing device having the above.
(2)
The head pose-rotation correlation unit stores the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired.
The information processing device according to (1) above.
(3)
a target display unit that displays a target that the user is paying attention to on a display;
a rotation detection unit that detects the eye rotation of the user accompanying the movement of the target;
and
the head pose estimation unit acquires the head pose of the user that varies in accordance with movement of the target;
the headpose-rotation correlation unit acquires a correlation between the eye rotation accompanying the movement of the target and the headpose as the headpose-rotation correlation;
The information processing device according to (2) above.
(4)
the target display unit notifies a message prompting the user to change the head pose.
The information processing device according to (3) above.
(5)
the target display unit displays the target at a position that causes a change in the head pose.
The information processing device according to (3) or (4) above.
(6)
the target display unit displays an obstruction in front of the target, and prompts the user to move the head pose in a manner that looks at the target behind the obstruction;
The information processing device according to (5) above.
(7)
the headpose-rotation correlator sequentially updates the headpose-rotation correlation based on data of the eye rotation and the headpose of the user intermittently acquired based on a preset trigger;
The information processing device according to any one of (2) to (6) above.
(8)
a scene detection unit that detects a video scene including an attention region as a peculiar scene;
a rotation detection unit that detects the eye rotation of the user accompanying the movement of the attraction region, using the detection of the peculiar scene as the trigger;
and
the head pose estimation unit acquires the head pose of the user that varies in accordance with the movement of the interest region;
the head pose-rotation correlation unit updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying the movement of the attention region and the head pose.
The information processing device according to (7) above.
(9)
the headpose-rotation correlation unit stores an average correlation between the eye rotation and the headpose based on data of a plurality of subjects as the headpose-rotation correlation;
The information processing device according to (1) above.
(10)
an optical axis estimation unit that acquires the optical axis of the eyeball,
The optical axis-visual axis correlation unit
Obtaining a correlation between the optical axis and the visual axis as an optical axis-visual axis correlation;
applying the optical axis to the optical axis-visual axis correlation to estimate the visual axis;
correcting the estimated visual axis based on the eye rotation;
The information processing device according to any one of (1) to (9) above.
(11)
The optical axis-visual axis correlation unit stores the correlation between the optical axis and the visual axis acquired for the user from whom the optical axis is to be acquired.
The information processing device according to (10) above.
(12)
Obtaining a correlation between eye rotation and head pose as a head pose-rotation correlation;
obtaining the head pose;
correcting the visual axis of the eye based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation;
1. A computer-implemented information processing method comprising:
(13)
storing the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired;
The information processing method according to (12) above.
(14)
Displaying a target that the user is paying attention to on a display;
detecting the eye rotation of the user accompanying the movement of the target;
Having that,
the head pose acquisition process acquires the head pose of the user that changes in accordance with the movement of the target;
The process of acquiring the head pose-rotation correlation includes acquiring a correlation between the eye rotation accompanying the movement of the target and the head pose as the head pose-rotation correlation.
The information processing method according to (13) above.
(15)
notifying a message prompting the head pose change;
The information processing method according to (14) above.
(16)
The target display process displays the target at a position that causes a change in the head pose.
The information processing method according to (14) or (15) above.
(17)
The target display process includes displaying an obstruction in front of the target, and prompting the user to move the head pose so as to look at the target behind the obstruction.
The information processing method according to (16) above.
(18)
The process of acquiring the head pose-rotation correlation includes sequentially updating the head pose-rotation correlation based on data of the eye rotation and the head pose of the user acquired intermittently based on a preset trigger.
The information processing method according to any one of (13) to (17) above.
(19)
The process of acquiring the head pose-rotation correlation includes storing an average correlation between the eye rotation and the head pose based on data of a plurality of subjects as the head pose-rotation correlation.
The information processing method according to (12) above.
(20)
Obtaining a correlation between eye rotation and head pose as a head pose-rotation correlation;
obtaining the head pose;
correcting the visual axis of the eye based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation;
A program that makes a computer do something.

　情報凊理装眮
　ヘッドポヌズ掚定郚
　ヘッドポヌズ回旋盞関郚
　タヌゲット衚瀺郚
　回旋怜出郚
　光軞掚定郚
　光軞芖軞盞関郚
　ディスプレむ
県球
　光軞
　遮蔜物
　タヌゲット
　ナヌザ
　芖軞 10, 30 Information processing device 12 Head pose estimation unit 13 Head pose-rotation correlation unit 14 Target display unit 16 Rotation detection unit 17 Optical axis estimation unit 18 Optical axis-visual axis correlation unit 23 Display EB Eyeball LA Optical axis OB Obstruction object TG Target US User VA Visual axis

Claims (20)

　県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗するヘッドポヌズ回旋盞関郚ず、
　前蚘ヘッドポヌズを取埗するヘッドポヌズ掚定郚ず、
　前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお埗られた前蚘県球回旋に基づいお県球の芖軞を補正する光軞芖軞盞関郚ず、
　を有する情報凊理装眮。 a headpose-rotation correlation unit that acquires a correlation between eye rotation and head pose as a headpose-rotation correlation;
a head pose estimation unit for acquiring the head pose;
an optical axis-visual axis correlation unit that corrects the visual axis of the eyeball based on the eyeball rotation obtained by applying the headpose to the headpose-rotation correlation;
An information processing device having the above. 　前蚘ヘッドポヌズ回旋盞関郚は、前蚘ヘッドポヌズの取埗察象ずなるナヌザに぀いお取埗された、前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を蚘憶する、
　請求項に蚘茉の情報凊理装眮。 The head pose-rotation correlation unit stores the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired.
The information processing device according to claim 1 . 　前蚘ナヌザが泚目するタヌゲットをディスプレむに衚瀺するタヌゲット衚瀺郚ず、
　前蚘タヌゲットの移動に䌎う前蚘ナヌザの前蚘県球回旋を怜出する回旋怜出郚ず、
　を有し、
　前蚘ヘッドポヌズ掚定郚は、前蚘タヌゲットの移動に䌎っお倉動する前蚘ナヌザの前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズ回旋盞関郚は、前蚘タヌゲットの移動に䌎う前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお取埗する、
　請求項に蚘茉の情報凊理装眮。 a target display unit that displays a target that the user is paying attention to on a display;
a rotation detection unit that detects the eye rotation of the user accompanying the movement of the target;
and
the head pose estimation unit acquires the head pose of the user that varies in accordance with movement of the target;
the headpose-rotation correlation unit acquires a correlation between the eye rotation accompanying the movement of the target and the headpose as the headpose-rotation correlation;
The information processing device according to claim 2 . 　前蚘タヌゲット衚瀺郚は、前蚘ヘッドポヌズの倉動を促すメッセヌゞを通知する、
　請求項に蚘茉の情報凊理装眮。 the target display unit notifies a message prompting the user to change the head pose.
The information processing device according to claim 3 . 　前蚘タヌゲット衚瀺郚は、前蚘ヘッドポヌズの倉動を匕き起こさせるような䜍眮に前蚘タヌゲットを衚瀺する、
　請求項に蚘茉の情報凊理装眮。 the target display unit displays the target at a position that causes a change in the head pose.
The information processing device according to claim 3 . 　前蚘タヌゲット衚瀺郚は、前蚘タヌゲットの前方に遮蔜物を衚瀺し、前蚘遮蔜物の裏偎にある前蚘タヌゲットを芗き蟌むような前蚘ヘッドポヌズの動きを促す、
　請求項に蚘茉の情報凊理装眮。 the target display unit displays an obstruction in front of the target, and prompts the user to move the head pose in a manner that looks at the target behind the obstruction;
The information processing device according to claim 5 . 　前蚘ヘッドポヌズ回旋盞関郚は、予め蚭定されたトリガに基づいお間欠的に取埗された前蚘ナヌザの前蚘県球回旋および前蚘ヘッドポヌズのデヌタに基づいお、前蚘ヘッドポヌズ回旋盞関を逐次曎新する、
　請求項に蚘茉の情報凊理装眮。 the headpose-rotation correlator sequentially updates the headpose-rotation correlation based on data of the eye rotation and the headpose of the user intermittently acquired based on a preset trigger;
The information processing device according to claim 2 . 　誘目領域を含む映像シヌンを特異シヌンずしお怜出するシヌン怜出郚ず、
　前蚘特異シヌンが怜出されたこずを前蚘トリガずしお、前蚘誘目領域の移動に䌎う前蚘ナヌザの前蚘県球回旋を怜出する回旋怜出郚ず、
　を有し、
　前蚘ヘッドポヌズ掚定郚は、前蚘誘目領域の移動に䌎っお倉動する前蚘ナヌザの前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズ回旋盞関郚は、前蚘誘目領域の移動に䌎う前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関に基づいお前蚘ヘッドポヌズ回旋盞関を曎新する、
　請求項に蚘茉の情報凊理装眮。 a scene detection unit that detects a video scene including an attention region as a peculiar scene;
a rotation detection unit that detects the eye rotation of the user accompanying the movement of the attraction region, using the detection of the peculiar scene as the trigger;
and
the head pose estimation unit acquires the head pose of the user that varies in accordance with the movement of the interest region;
the head pose-rotation correlation unit updates the head pose-rotation correlation based on the correlation between the eye rotation accompanying the movement of the attention region and the head pose.
The information processing device according to claim 7 . 　前蚘ヘッドポヌズ回旋盞関郚は、耇数の被隓者のデヌタに基づく平均的な前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお蚘憶する、
　請求項に蚘茉の情報凊理装眮。 the headpose-rotation correlation unit stores, as the headpose-rotation correlation, a correlation between an average of the eye rotation and the headpose based on data of a plurality of subjects;
The information processing device according to claim 1 . 　前蚘県球の光軞を取埗する光軞掚定郚を有し、
　前蚘光軞芖軞盞関郚は、
　　前蚘光軞ず前蚘芖軞ずの間の盞関を光軞芖軞盞関ずしお取埗し、
　　前蚘光軞を前蚘光軞芖軞盞関に適甚しお前蚘芖軞を掚定し、
　　掚定された前蚘芖軞を前蚘県球回旋に基づいお補正する、
　請求項に蚘茉の情報凊理装眮。 an optical axis estimation unit that acquires the optical axis of the eyeball,
The optical axis-visual axis correlation unit
Obtaining a correlation between the optical axis and the visual axis as an optical axis-visual axis correlation;
applying the optical axis to the optical axis-visual axis correlation to estimate the visual axis;
correcting the estimated visual axis based on the eye rotation;
The information processing device according to claim 1 . 　前蚘光軞芖軞盞関郚は、前蚘光軞の取埗察象ずなるナヌザに぀いお取埗された、前蚘光軞ず前蚘芖軞ずの間の盞関を蚘憶する、
　請求項に蚘茉の情報凊理装眮。 The optical axis-visual axis correlation unit stores the correlation between the optical axis and the visual axis acquired for the user from whom the optical axis is to be acquired.
The information processing device according to claim 10. 　県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗し、
　前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお埗られた前蚘県球回旋に基づいお県球の芖軞を補正する、
　こずを有する、コンピュヌタにより実行される情報凊理方法。 Obtaining a correlation between eye rotation and head pose as a head pose-rotation correlation;
obtaining the head pose;
correcting the visual axis of the eye based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation;
10. A computer-implemented information processing method comprising: 　前蚘ヘッドポヌズの取埗察象ずなるナヌザに぀いお取埗された、前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を蚘憶するこずを有する、
　請求項に蚘茉の情報凊理方法。 storing the correlation between the eye rotation and the head pose acquired for the user whose head pose is to be acquired;
The information processing method according to claim 12. 　前蚘ナヌザが泚目するタヌゲットをディスプレむに衚瀺し、
　前蚘タヌゲットの移動に䌎う前蚘ナヌザの前蚘県球回旋を怜出する、
　こずを有し、
　前蚘ヘッドポヌズの取埗凊理は、前蚘タヌゲットの移動に䌎っお倉動する前蚘ナヌザの前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズ回旋盞関の取埗凊理は、前蚘タヌゲットの移動に䌎う前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお取埗する、
　請求項に蚘茉の情報凊理方法。 Displaying a target that the user is paying attention to on a display;
detecting the eye rotation of the user accompanying the movement of the target;
Having that,
the head pose acquisition process acquires the head pose of the user that changes in accordance with the movement of the target;
The process of acquiring the head pose-rotation correlation includes acquiring a correlation between the eye rotation accompanying the movement of the target and the head pose as the head pose-rotation correlation.
The information processing method according to claim 13. 　前蚘ヘッドポヌズの倉動を促すメッセヌゞを通知するこずを有する、
　請求項に蚘茉の情報凊理方法。 notifying a message prompting the head pose change;
The information processing method according to claim 14. 　前蚘タヌゲットの衚瀺凊理は、前蚘ヘッドポヌズの倉動を匕き起こさせるような䜍眮に前蚘タヌゲットを衚瀺する、
　請求項に蚘茉の情報凊理方法。 The target display process displays the target at a position that causes a change in the head pose.
The information processing method according to claim 14. 　前蚘タヌゲットの衚瀺凊理は、前蚘タヌゲットの前方に遮蔜物を衚瀺し、前蚘遮蔜物の裏偎にある前蚘タヌゲットを芗き蟌むような前蚘ヘッドポヌズの動きを促す、
　請求項に蚘茉の情報凊理方法。 The target display process includes displaying an obstruction in front of the target, and prompting the user to move the head pose so as to look at the target behind the obstruction.
17. The information processing method according to claim 16. 　前蚘ヘッドポヌズ回旋盞関の取埗凊理は、予め蚭定されたトリガに基づいお間欠的に取埗された前蚘ナヌザの前蚘県球回旋および前蚘ヘッドポヌズのデヌタに基づいお、前蚘ヘッドポヌズ回旋盞関を逐次曎新する、
　請求項に蚘茉の情報凊理方法。 The process of acquiring the head pose-rotation correlation includes sequentially updating the head pose-rotation correlation based on data of the eye rotation and the head pose of the user acquired intermittently based on a preset trigger.
The information processing method according to claim 13. 　前蚘ヘッドポヌズ回旋盞関の取埗凊理は、耇数の被隓者のデヌタに基づく平均的な前蚘県球回旋ず前蚘ヘッドポヌズずの間の盞関を前蚘ヘッドポヌズ回旋盞関ずしお蚘憶する、
　請求項に蚘茉の情報凊理方法。 The process of acquiring the head pose-rotation correlation includes storing an average correlation between the eye rotation and the head pose based on data of a plurality of subjects as the head pose-rotation correlation.
The information processing method according to claim 12. 　県球回旋ずヘッドポヌズずの間の盞関をヘッドポヌズ回旋盞関ずしお取埗し、
　前蚘ヘッドポヌズを取埗し、
　前蚘ヘッドポヌズを前蚘ヘッドポヌズ回旋盞関に適甚しお埗られた前蚘県球回旋に基づいお県球の芖軞を補正する、
　こずをコンピュヌタに実珟させるプログラム。 Obtaining a correlation between eye rotation and head pose as a head pose-rotation correlation;
obtaining the head pose;
correcting the visual axis of the eye based on the eye rotation obtained by applying the head pose to the head pose-rotation correlation;
A program that makes a computer do something.
PCT/JP2025/013086 2024-04-10 2025-03-31 Information processing device, information processing method, and program Pending WO2025216110A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2024063170 2024-04-10
JP2024-063170 2024-04-10

Publications (1)

Publication Number Publication Date
WO2025216110A1 true WO2025216110A1 (en) 2025-10-16

Family

ID=97349821

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2025/013086 Pending WO2025216110A1 (en) 2024-04-10 2025-03-31 Information processing device, information processing method, and program

Country Status (1)

Country Link
WO (1) WO2025216110A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015169959A (en) * 2014-03-04 2015-09-28 囜立倧孊法人静岡倧孊 Rotation angle calculation method, gazing point detection method, information input method, rotation angle calculation device, gazing point detection device, information input device, rotation angle calculation program, gazing point detection program, and information input program
JP2016198387A (en) * 2015-04-13 2016-12-01 株匏䌚瀟クリュヌトメディカルシステムズ Visual inspection device, visual target correction method for visual inspection device, and display device
JP2017129898A (en) * 2016-01-18 2017-07-27 ゜ニヌ株匏䌚瀟 Information processing apparatus, information processing method, and program
JP2021069008A (en) * 2019-10-23 2021-04-30 キダノン株匏䌚瀟 Electronic equipment, control method for electronic equipment, program and storage medium
US20210349528A1 (en) * 2018-10-24 2021-11-11 Pcms Holdings, Inc. Systems and methods for region of interest estimation for virtual reality

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015169959A (en) * 2014-03-04 2015-09-28 囜立倧孊法人静岡倧孊 Rotation angle calculation method, gazing point detection method, information input method, rotation angle calculation device, gazing point detection device, information input device, rotation angle calculation program, gazing point detection program, and information input program
JP2016198387A (en) * 2015-04-13 2016-12-01 株匏䌚瀟クリュヌトメディカルシステムズ Visual inspection device, visual target correction method for visual inspection device, and display device
JP2017129898A (en) * 2016-01-18 2017-07-27 ゜ニヌ株匏䌚瀟 Information processing apparatus, information processing method, and program
US20210349528A1 (en) * 2018-10-24 2021-11-11 Pcms Holdings, Inc. Systems and methods for region of interest estimation for virtual reality
JP2021069008A (en) * 2019-10-23 2021-04-30 キダノン株匏䌚瀟 Electronic equipment, control method for electronic equipment, program and storage medium

Similar Documents

Publication Publication Date Title
US20230069764A1 (en) Systems and methods for using natural gaze dynamics to detect input recognition errors
US10534982B2 (en) Neural network training for three dimensional (3D) gaze prediction with calibration parameters
US10241329B2 (en) Varifocal aberration compensation for near-eye displays
US20200250488A1 (en) Deep learning for three dimensional (3d) gaze prediction
CN110249337A (en) Using eye tracks camera to facial expression classification
US20180068449A1 (en) Sensor fusion systems and methods for eye-tracking applications
CN110320998B (en) Training of neural networks for 3D gaze prediction
US10976816B2 (en) Using eye tracking to hide virtual reality scene changes in plain sight
JP7367689B2 (en) Information processing device, information processing method, and recording medium
WO2017074807A1 (en) Adjusting image frames based on tracking motion of eyes
WO2017074662A1 (en) Tracking of wearer's eyes relative to wearable device
CN114391117A (en) Eye tracking delay enhancement
US11054903B2 (en) Method, eyetracker and computer program for determining eye positions in digital image data
US20190324528A1 (en) Adjusting gaze point based on determined offset adjustment
US12028419B1 (en) Systems and methods for predictively downloading volumetric data
CN109451240A (en) Focusing method, device, computer equipment and readable storage medium storing program for executing
US20230053497A1 (en) Systems and methods for performing eye-tracking
CN120161953B (en) Imaging display method and system for head-mounted display device
WO2025216110A1 (en) Information processing device, information processing method, and program
JP7423005B2 (en) Calibration method and device for line of sight measurement using changes in pupil diameter, line of sight measurement device and camera device
US20250175585A1 (en) Binocular disparity correction system
CN115268650A (en) Picture screen capturing method and device, head-mounted virtual reality equipment and storage medium
CN113780414A (en) Eye movement behavior analysis method, image rendering method, component, device and medium
JP7442300B2 (en) Playback control device and playback control program
US20250373943A1 (en) Multi-Modal Sensor Fusion for Camera Focus Adjustments

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25786393

Country of ref document: EP

Kind code of ref document: A1