[go: up one dir, main page]

WO2021029256A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et programme - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et programme Download PDF

Info

Publication number
WO2021029256A1
WO2021029256A1 PCT/JP2020/029699 JP2020029699W WO2021029256A1 WO 2021029256 A1 WO2021029256 A1 WO 2021029256A1 JP 2020029699 W JP2020029699 W JP 2020029699W WO 2021029256 A1 WO2021029256 A1 WO 2021029256A1
Authority
WO
WIPO (PCT)
Prior art keywords
information processing
display
indicator
processing device
orientation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/029699
Other languages
English (en)
Japanese (ja)
Inventor
貴光 後藤
西堀　一彦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Publication of WO2021029256A1 publication Critical patent/WO2021029256A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B30/00—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/37—Details of the operation on graphic patterns
    • G09G5/377—Details of the operation on graphic patterns for mixing or overlaying two or more graphic patterns
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/361—Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking

Definitions

  • This technology relates to information processing devices, information processing methods, and programs used for input operations.
  • Non-Patent Document 1 describes a pen-type input device (stylus) whose tip expands and contracts.
  • a telescopic shaft is housed inside the grip.
  • the stylus is also equipped with an acceleration sensor and a magnetic sensor for calculating its orientation. For example, when the shaft is pressed against the surface of a touch screen, a virtual stylus is drawn in the digital space inside the screen along the axis of the actual stylus. This makes it possible to perform a three-dimensional input operation on an object in the digital space.
  • the purpose of this technology is to provide an information processing device, an information processing method, and a program capable of realizing an intuitive input operation for a stereoscopic image.
  • the information processing apparatus includes a detection unit and a display control unit.
  • the detection unit detects the position and orientation of the indicator based on the image of the indicator used by the user.
  • the display control unit causes the display space of the stereoscopic display device to display an instruction object representing a virtual insertion portion of the indicator with respect to the display space, based on the position and orientation of the indicator.
  • the position and orientation of the indicator are detected from the image of the indicator used by the user. Based on this detection result, the virtual insertion portion of the indicator into the display space of the stereoscopic display device is displayed as an instruction object. As a result, it becomes possible to easily operate the instruction object in the display space by using the indicator, and it is possible to realize an intuitive input operation for the stereoscopic image.
  • the display control unit may calculate a virtual insertion amount of the indicator into the display space based on the position and orientation of the indicator, and control the display of the instruction object according to the insertion amount. ..
  • the display control unit may control the display of the instruction object so that the length of the instruction object is proportional to the insertion amount.
  • the stereoscopic display device may have a display panel for displaying a stereoscopic image.
  • the display space may be a virtual space with the display panel as a boundary.
  • the detection unit may detect the position and orientation of the indicator with respect to the display panel.
  • the indicator may expand and contract in one direction in response to pressing.
  • the display control unit may calculate the amount of expansion and contraction of the indicator in response to the contact between the display panel and the indicator as a virtual insertion amount of the indicator into the display space.
  • the display control unit may display the instruction object starting from the contact position between the display panel and the indicator in the display space.
  • the stereoscopic display device may have an imaging unit directed to the observation range of the display panel.
  • the detection unit may detect the position and orientation of the indicator used by the user in the observation range based on the image of the observation range captured by the imaging unit.
  • the detection unit may detect the viewpoint of the user who observes the display panel based on the image of the observation range captured by the imaging unit.
  • the display control unit may display the stereoscopic image according to the viewpoint of the user.
  • the indicator may have a marker portion.
  • the detection unit may detect the position and orientation of the marker unit as the position and orientation of the indicator.
  • the indicator may have a tip portion directed at the target and a grip portion connected to the tip portion.
  • the marker portion may be connected to the side opposite to the side to which the tip portion of the grip portion is connected.
  • the display control unit may display a target object to be an input operation using the indicator in the display space.
  • the display control unit may set the type of the instruction object according to the type of the target object.
  • the display control unit may control the display of the target object according to the contact between the target object and the instruction object.
  • the indicator may have a sensation presenting unit that presents a skin sensation to the user.
  • the information processing device may further include a sensory control unit that controls the sensory presentation unit in response to contact between the target object and the instruction object.
  • the sensation presenting unit may present at least one of vibration sensation and thermal sensation.
  • the sensory control unit may control the sensory presentation unit according to at least one of the type of the target object, the type of the instruction object, or the amount of contact between the target object and the instruction object.
  • the information processing method is an information processing method executed by a computer system, and includes detecting the position and orientation of the indicator based on an image of the indicator used by the user. Based on the position and orientation of the indicator, an instruction object representing a virtual insertion portion of the indicator with respect to the display space is displayed in the display space of the stereoscopic display device.
  • a program causes a computer system to perform the following steps.
  • FIG. 1 is a schematic diagram for explaining an outline of the present technology.
  • the present technology can realize an input operation for the stereoscopic image 10 displayed by the stereoscopic display device 100, and can provide feedback according to the input operation.
  • FIG. 1A is a schematic diagram showing a state before the input operation of the user 1 is performed on the stereoscopic image 10
  • FIG. 1B is a schematic diagram showing a state in which the input operation of the user 1 is performed.
  • 1A and 1B schematically show a user 1 observing a stereoscopic image 10 (rabbit object) displayed by the stereoscopic display device 100.
  • the three-dimensional display device 100 is a device that three-dimensionally displays content or the like to be viewed.
  • the stereoscopic display device 100 performs stereoscopic display (stereoscopic stereoscopic vision) using binocular parallax by displaying, for example, an image for the right eye and an image for the left eye.
  • stereoscopic display stereoscopic stereoscopic vision
  • the user 1 observing the stereoscopic display device 100 can perceive the target content as a stereoscopic image 10 having a depth.
  • the space in which the stereoscopic image 10 is displayed will be referred to as the display space 11 of the stereoscopic display device 100.
  • the display space 11 is, for example, a three-dimensional virtual space that extends in the vertical direction, the horizontal direction, and the depth direction. Therefore, the stereoscopic image 10 can be said to be a virtual object that is stereoscopically displayed in the display space 11.
  • the stereoscopic image 10 is displayed by the display panel 20 of the stereoscopic display device 100.
  • a display panel 20 arranged obliquely with respect to the horizontal direction (vertical direction) is schematically illustrated as a shaded area.
  • the display panel 20 is a display that displays an image for the right eye and an image for the left eye on the right eye (right viewpoint) and the left eye (left viewpoint) of the user 1, respectively.
  • the user 1 can perceive the stereoscopic image 10 simply by looking at the display panel 20.
  • the stereoscopic display device 100 is a naked-eye stereoscopic display device that displays the stereoscopic image 10 to the user 1 in the state of the naked eye without using a glasses-type device or the like. Further, in the stereoscopic display device 100, the position of the viewpoint 2 of the user 1 is tracked by using a camera 21 (for example, a monocular RGB camera) arranged in the vicinity of the display panel 20, and the right according to the position of the viewpoint 2 of the user 1. An eye image and a left eye image are generated. As a result, even when the position of the viewpoint 2 is moved, it is possible to perceive the stereoscopic image 10 (rabbit object or the like) seen from the position of the viewpoint 2 after the movement.
  • a camera 21 for example, a monocular RGB camera
  • the display space 11 of the stereoscopic display device 100 is a virtual space whose boundary is the display panel 20.
  • FIG. 1A schematically illustrates a rabbit object (stereoscopic image 10) displayed in the display space 11 inside the display panel 20.
  • the rabbit object is an example of the target object 14.
  • the target object 14 is an object (stereoscopic image 10) that is the target of the input operation of the user 1.
  • the pointing device 40 is used for the input operation on the target object 14.
  • the pointing device 40 is a rod-shaped device configured to be expandable and contractible (see FIG. 2).
  • the user 1 who has grasped the pointing device 40 can perform an input operation on the target object 14 by operating the tip of the pointing device 40 so as to approach the target object 14.
  • the pointing device 40 is provided with a marker 41 for detecting the position and orientation thereof.
  • FIG. 1A schematically illustrates the pointing device 40 used by the user 1.
  • the pointing device 40 corresponds to an indicator.
  • the marker 41 corresponds to a marker portion.
  • FIG. 1B shows how the user 1 uses the pointing device 40 to perform an input operation on the rabbit object, which is the target object 14. It is assumed that the user 1 brings the tip of the pointing device 40 close to the target object 14. At this time, the camera 21 captures a two-dimensional image of the marker 41 of the pointing device 40, and the three-dimensional position and orientation of the marker 41 are estimated based on the captured two-dimensional image. As a result, the position and orientation of the pointing device 40, that is, from which position to which direction the pointing device 40 is directed is estimated.
  • the virtual pointer 15 is displayed in the display space 11 in the display panel 20.
  • the virtual pointer 15 is a stereoscopic image 10 displayed in the display space 11, and is a virtual object representing a virtual insertion portion of the pointing device 40 in the display space 11.
  • the virtual pointer 15 corresponds to an instruction object.
  • the virtual pointer 15 corresponds to the insertion amount (expansion / contraction amount) of the pointing device 40 from the contact point between the pointing device 40 and the display panel 20 along the direction in which the pointing device 40 is directed. It is displayed so that it is the length. That is, it can be said that the virtual pointer 15 is an object in which the pointing device 40 used by the user 1 in the real space is extended to the virtual space (display space 11) and displayed.
  • the pointing device 40 For example, if the pointing device 40 is moved while the pointing device 40 is in contact with the display panel 20, the contact point moves and the virtual pointer 15 moves. Further, when the orientation of the pointing device 40 is changed, the orientation of the virtual pointer 15 changes along the orientation of the pointing device 40. Further, when the insertion amount (expansion / contraction amount) of the pointing device 40 is changed, the length of the virtual pointer 15 changes. Therefore, the user 1 can intuitively operate the virtual pointer 15 in the virtual space (display space 11) by operating the pointing device 40 which is an object in the real space.
  • the rabbit object which is the target object 14
  • the rabbit object is displayed so as to look in the direction in which the virtual pointer 15 is in contact.
  • the pointing device 40 is equipped with a vibration presenting unit (see FIG. 2) that generates vibration. For example, when the virtual pointer 15 and the target object 14 come into contact with each other, vibration of a predetermined vibration pattern is presented via the vibration presenting unit. As a result, the user 1 can perceive the sensation of contact with the target object 14 as vibration. By vibrating the pointing device 40 in this way, it is possible to feed back to the user 1 the feeling as if the object object 14 which is the stereoscopic image 10 was actually touched.
  • the operation of bringing the virtual pointer 15 into contact with the target object 14 is an example of an input operation for the target object 14 by the user 1.
  • the display of the target object 14 changes according to the contact with the virtual pointer 15, and the feeling of contact is fed back to the user 1 as vibration.
  • the user 1 can perform an intuitive input operation on the target object 14 displayed by the stereoscopic display device 100 by using the existing pointing device 40.
  • vibration is fed back according to the state of operation, it is possible to provide an experience as if the operation is being applied to a real object.
  • FIG. 2 is a schematic view showing a configuration example of the pointing device 40.
  • 2A and 2B schematically show a state in which the pointing device 40 is extended and a state in which the pointing device 40 is contracted.
  • the pointing device 40 has an expansion / contraction portion 42, a grip portion 43, and a marker 41. Further, inside the pointing device 40, a communication unit 44 and a vibration presentation unit 45 are provided.
  • the telescopic portion 42 has a rod-shaped structure extending in one direction as a whole, and is a portion directed to the target. One end of the telescopic portion 42 is connected to the grip portion 43, and the other end is a tip 46 directed at the target.
  • the direction in which the telescopic portion 42 extends is referred to as an axial direction.
  • the telescopic portion 42 corresponds to the tip portion directed to the target, and the axial direction corresponds to one direction.
  • the telescopic portion 42 is configured to contract in the axial direction when the tip 46 is pushed.
  • the telescopic portion 42 for example, a telescopic structure in which a plurality of cylinders arranged in a nested manner along the axial direction expand and contract is used.
  • the stretchable portion may be configured using any stretchable structure.
  • the grip portion 43 is a tubular longitudinal member connected to the telescopic portion 42, and is a portion gripped by the user 1 who uses the pointing device 40.
  • the outermost cylinder in the telescopic portion 42 of the telescopic structure is used as the grip portion 43.
  • the marker 41 has a plurality of detection points 47 used for detecting the position and orientation.
  • the position and orientation of the marker 41 can be detected from the positional relationship of each detection point 47 displayed in the image obtained by capturing the marker 41.
  • the marker 41 is typically provided with three or more detection points 47.
  • the detection point 47 for example, a vertex of a solid, a coloring point or a light emitting point having different colors from each other is used.
  • the marker 41 is connected to the side opposite to the side to which the telescopic portion 42 of the grip portion 43 is connected. That is, the marker 41 is provided at the end opposite to the tip 46 directed to the target. This makes it possible to properly detect the marker 41.
  • the marker 41 on the pentagon is schematically illustrated. In this case, for example, each vertex of the pentagon becomes the detection point 47.
  • the specific configuration of the marker 41 is not limited.
  • the communication unit 44 and the vibration presentation unit 45 are provided inside, for example, the grip unit 43 (or the marker 41).
  • the communication unit 44 is a communication module for communicating with other devices, and communicates with, for example, the stereoscopic display device 100.
  • a wireless LAN (Local Area Network) module such as WiFi or a communication module for short-range wireless communication such as Bluetooth (registered trademark) is used.
  • the vibration presenting unit 45 presents the vibration sensation to the user 1.
  • a vibration element such as a voice coil motor, a linear actuator, an eccentric motor, and a piezo element is used.
  • the vibration presenting unit 45 when the vibration presenting unit 45 generates vibration based on a predetermined vibration signal, the vibration sensation corresponding to the vibration signal is presented to the user 1 via the grip unit 43. This makes it possible for the user 1 to perceive various tactile sensations (haptics).
  • the vibration presenting unit 45 functions as a sensation presenting unit that presents a skin sensation to the user.
  • a sensation presenting unit for presenting a thermal sensation to the user 1 may be mounted.
  • the heat presenting unit for example, a heater element, a Peltier element, or the like is used. By installing a heat presenting unit, it is possible to present a hot sensation or a cold sensation.
  • the pointing device 40 may be equipped with an element (sensory presentation unit) capable of presenting any skin sensation such as tactile sensation, heat sensation, and force sensation.
  • FIG. 2B shows a state in which the pointing device 40 is pressed against the display panel 20 and contracted.
  • the telescopic portion 42 contracts when the tip 46 is pushed by the display panel 20.
  • the cylinders provided on the tip 46 side of the telescopic portion 42 are pushed toward the grip portion 43 in order from the side closer to the tip 46.
  • FIG. 2B schematically shows a cylinder pushed toward the grip portion 43 side.
  • the telescopic portion 42 is provided with a restoring mechanism (not shown) that generates a restoring force that pushes back the tip 46 when the tip 46 is pushed.
  • the specific configuration of the restoration mechanism is not limited, and for example, an elastic body such as a spring or rubber, a pressure cylinder, or the like is used.
  • the tip 46 of the telescopic portion 42 is pushed back and extended by the restoring mechanism. Therefore, while the tip 46 is pressed against the object (for example, the display panel 20), the telescopic portion 42 is contracted, but when the tip 46 is not pressed, the telescopic portion 42 is as shown in FIG. 2A. It returns to the stretched state.
  • the pointing device 40 can be expanded and contracted in the axial direction in response to pressing. That is, the pointing device 40 is configured such that when it comes into contact with the display panel 20, the tip portion is reduced by the expansion / contraction mechanism (expansion / contraction portion). As will be described later, the expansion / contraction amount of the pointing device 40 is used for display control of the virtual pointer 15 as a virtual insertion amount of the pointing device 40 into the display panel 20. For example, the reduced tip is displayed as the virtual pointer 15. As a result, an intuitive input operation as if the pointing device 40 is inserted as it is into the display space 11 in the display panel 20 becomes possible. In addition, input can be intuitively added to the target object 14 localized on the back side of the surface of the display panel 20 via the virtual pointer 15.
  • FIG. 3 is a block diagram showing a functional configuration example of the three-dimensional display device 100.
  • the stereoscopic display device 100 includes a display panel 20, a camera 21, a speaker 22, an operation unit 23, an I / F (interface) unit 24, a communication unit 25, and a storage unit 26 described with reference to FIG. , And a controller 27.
  • the display panel 20 displays the stereoscopic image 10. Specifically, the right eye image and the left eye image capable of stereoscopic stereoscopic viewing are displayed on the right eye and the left eye of the user 1, respectively.
  • the display panel 20 emits light rays forming an image for the right eye toward a certain direction of the right eye, and emits light rays forming an image for the left eye toward a certain direction of the left eye. This makes it possible to give the user 1 binocular parallax to perceive the stereoscopic image 10.
  • the light rays for perceiving the stereoscopic image 10 (right eye image and left eye image) are emitted toward a predetermined observation range set in front of the display panel 20. Therefore, it can be said that the display panel 20 is a device that allows the user 1 who observes the display panel 20 from the observation range to perceive the stereoscopic image 10.
  • the display panel 20 is, for example, a rectangular panel in a plan view. In the present embodiment, the display panel 20 is arranged at an angle when viewed from the user 1 so that the pair of sides are parallel to the horizontal direction. Since the surface of the display panel 20 is arranged at an angle in this way, the user 1 can observe the stereoscopic image 10 from the horizontal direction and the vertical direction, and the observation range is expanded.
  • the method of the display panel 20 is not limited. For example, a parallax barrier method in which a shielding plate is provided for each set of display pixels to separate light rays incident on each eye, a lenticular lens method in which the emission direction is controlled for each display pixel, and the like are used. Further, a light ray reproduction type display or the like may be used. In addition, any method capable of stereoscopic display with the naked eye may be adopted.
  • the camera 21 is arranged toward the observation range of the display panel 20.
  • the camera 21 is arranged at the center of the upper end of the display panel 20 so as to capture the space (observation range) in front of the display panel 20 arranged diagonally.
  • the camera 21 corresponds to an imaging unit.
  • the camera 21 is typically a monocular RGB camera capable of capturing color moving and still images. The present technology can be applied even when a plurality of cameras 21 such as a stereo camera 21 are used.
  • a digital camera including an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor can be used.
  • any configuration may be adopted.
  • the speaker 22 can output various sounds.
  • the specific configuration of the speaker 22 is not limited.
  • the operation unit 23 is an operation device that performs physical operations such as a keyboard and a mouse.
  • the I / F unit 24 is an interface to which other devices and various cables are connected, such as a USB (Universal Serial Bus) terminal and an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.
  • the communication unit 25 is a communication module for communicating with another device, and communicates with, for example, the pointing device 40.
  • a wireless LAN module such as WiFi
  • a communication module for short-range wireless communication such as Bluetooth (registered trademark) is used.
  • the storage unit 26 is a non-volatile storage device, and for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like is used.
  • the object library 28 is stored in the storage unit 26.
  • the object library 28 is, for example, a library in which the virtual pointer 15 and the design data of the object to be the target object 14 are stored. For example, when displaying the virtual pointer 15, the target object 14, or the like, the design data stored in the object library 28 is appropriately read out.
  • the storage unit 26 stores a control program 29 for controlling the overall operation of the stereoscopic display device 100.
  • the control program 29 includes a program related to the present technology. The method of installing the control program 29 on the stereoscopic display device 100 is not limited.
  • the installation may be executed via various recording media, or the program may be installed via the Internet or the like.
  • the type of recording medium on which the program according to the present technology is recorded is not limited, and any computer-readable recording medium may be used.
  • any recording medium for recording data non-temporarily may be used.
  • the controller 27 controls the operation of each block included in the stereoscopic display device 100.
  • the controller 27 has hardware necessary for configuring a computer, such as a processor such as a CPU or GPU and a memory such as ROM or RAM.
  • the information processing method according to the present technology is executed by loading and executing the control program (program according to the present technology) 29 recorded in the storage unit 26 by the CPU or the like into the RAM.
  • the controller 27 corresponds to an information processing device.
  • the specific configuration of the controller 27 is not limited, and any hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit) may be used.
  • the CPU of the controller 27 or the like executes the program according to the present embodiment to perform the image acquisition unit 30, the viewpoint detection unit 31, the pointer detection unit 32, the display control unit 33, and the vibration control as functional blocks. Part 34 is realized. Then, the information processing method according to the present embodiment is executed by these functional blocks. In addition, in order to realize each functional block, dedicated hardware such as an IC (integrated circuit) may be appropriately used. In the present embodiment, the viewpoint detection unit 31 and the pointer detection unit 32 function as detection units.
  • the image acquisition unit 30 acquires an image of the observation range captured by the camera 21 (hereinafter, simply referred to as an captured image). For example, the camera 21 reads the captured images generated at a predetermined frame rate.
  • the captured image acquired by the image acquisition unit 30 is output to the viewpoint detection unit 31 and the pointer detection unit 32, which will be described later, respectively.
  • the viewpoint detection unit 31 detects the viewpoint 2 of the user 1 who observes the display panel 20 based on the captured image captured by the camera 21. More specifically, the viewpoint detection unit 31 detects the position of the viewpoint 2 of the user 1. For example, the face recognition of the user 1 observing the display panel 20 is executed, and the three-dimensional coordinates of the viewpoint position of the user 1 in the observation range are calculated.
  • the method for detecting the viewpoint 2 is not limited, and for example, a viewpoint estimation process using machine learning or the like, or a viewpoint detection using pattern matching or the like may be executed.
  • the pointer detection unit 32 detects the position and orientation of the pointing device 40 based on the image of the pointing device 40 used by the user 1. As described above, since the pointing device 40 is provided with the marker 41 for detecting the position and orientation, it is possible to easily detect the position and orientation of the pointing device 40 from the two-dimensional image. As the position of the pointing device 40, a three-dimensional position in a predetermined coordinate system is detected. Further, as the posture of the pointing device 40, an angle parameter representing the direction of the pointing device 40 (axial direction) in a predetermined coordinate system is detected. The method of detecting the position and orientation of the pointing device 40 (marker 41) will be described in detail later.
  • the observation image read by the image acquisition unit 30 is used as the image of the pointing device 40. That is, the pointer detection unit 32 detects the position and orientation of the pointing device 40 used by the user 1 in the observation range based on the captured image captured by the camera 21. That is, it can be said that the stereoscopic display device 100 detects the position and orientation of the pointing device 40 used by the user 1 by using the camera 21 that tracks the viewpoint 2 of the user 1. As a result, it is possible to detect the position and orientation of the pointing device 40 without adding a special sensor or the like, and it is possible to reduce the cost of the device or the like.
  • the case is not limited to the case where the camera 21 for viewpoint detection is used, and for example, an image obtained by the pointing device 40 by another imaging device may be used. Further, both the camera 21 for viewpoint detection and the image captured by another imaging device may be used. As a result, it is possible to improve the detection accuracy of the position and orientation and expand the detection range.
  • the display control unit 33 displays the stereoscopic image 10 according to the viewpoint 2 of the user 1.
  • the object displayed as the stereoscopic image 10 includes the target object 14 and the virtual pointer 15.
  • the display control unit 33 calculates the direction in which the object is observed based on the position of the viewpoint 2 of the user 1 detected by the viewpoint detection unit 31.
  • a parallax image (right eye image and left eye image) for displaying the object viewed from the calculated direction as a stereoscopic image 10 is generated, and the image data of the parallax image is output to the display panel 20.
  • the viewpoint 2 of the user 1 changes, it is possible to perceive the stereoscopic image 10 of the object viewed from that direction.
  • the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display the target object 14 that is the target of the input operation using the pointing device 40.
  • the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display the target object 14 that is the target of the input operation using the pointing device 40.
  • the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display the target object 14 that is the target of the input operation using the pointing device 40.
  • the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display the target object 14 that is the target of the input operation using the pointing device 40.
  • the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display the target object 14 that is the target of the input operation using the pointing device 40.
  • a parallax image for displaying the rabbit object as a stereoscopic image 10 is generated and output to the display panel 20.
  • the display control unit 33 causes the display space 11 of the stereoscopic display device 100 to display a virtual pointer 15 representing a virtual insertion portion of the pointing device 40 with respect to the display space 11 based on the position and orientation of the pointing device 40. More specifically, display parameters such as the position, posture, and length of the virtual pointer 15 are calculated based on the position and orientation of the pointing device 40 detected by the pointer detection unit 32. Based on this display parameter, a parallax image for displaying the virtual pointer 15 viewed from the viewpoint 2 of the user 1 as a stereoscopic image 10 is generated and output to the display panel 20. The target object 14 and the virtual pointer 15 are generated by using the design data stored in the object library 28.
  • the vibration control unit 34 controls the vibration presentation unit 45 provided in the pointing device 40 in response to the contact between the target object 14 and the virtual pointer 15. For example, when the target object 14 and the virtual pointer 15 come into contact with each other, the vibration control unit 34 generates a predetermined vibration signal and outputs it to the vibration presenting unit 45 via the communication units 25 and 44. As a result, at the timing when the virtual pointer 15 comes into contact with the target object 14, the user 1 holding the pointing device 40 is presented with a predetermined vibration sensation. If a heat presentation unit or the like is provided in addition to the vibration presentation unit 45, a heat control unit or the like that controls the heat presentation unit may be configured.
  • the heat control unit generates a control signal for controlling the temperature of the heat presentation unit and the like in response to the contact between the target object 14 and the virtual pointer 15. This makes it possible to present the user 1 with a predetermined thermal sensation.
  • the vibration control unit 34 heat presentation unit corresponds to the sensory control unit.
  • FIG. 4 is a flowchart showing an example of the information processing method according to the present embodiment.
  • FIG. 5 is a schematic diagram for explaining the arrangement relationship between the display panel 20 and the pointing device 40.
  • the display processing of the virtual pointer 15 will be described with reference to FIGS. 4 and 5.
  • the display control unit 33 calculates the virtual insertion amount of the pointing device 40 into the display space 11 based on the position and orientation of the pointing device 40. Then, the display of the virtual pointer 15 is controlled according to the calculated insertion amount.
  • the virtual insertion amount is the length at which the user 1 pushes the pointing device 40 into the display panel 20, and is the amount of expansion and contraction of the pointing device 40, as described with reference to FIG. 1B. The larger the expansion / contraction amount, the longer the length of the virtual pointer 15. For example, the user 1 operates the grip portion 43 so that the tip 46 of the pointing device 40 is brought closer to the target object 14. At this time, a virtual pointer 15 that moves in conjunction with the pointing device 40 is displayed inside the display panel 20.
  • the user 1 can bring the virtual pointer 15 closer to or in contact with a desired point on the target object 14 by adjusting the position and orientation of the pointing device 40 and the virtual pointer 15 which is an extension thereof. ..
  • the target object 14 displayed in the display space 11 is displayed by the visual information of the real or virtual displayed pointers (pointing device 40 and virtual pointer 15) according to the state of the pointer.
  • An input operation that gives input is realized.
  • a specific description will be given.
  • FIG. 5A is a perspective view showing a user 1 who performs an input operation on the stereoscopic display device 100 (display panel 20).
  • FIG. 5B is a top view of the display panel 20 when viewed from the front.
  • FIG. 5C is a side view of the display panel 20 when viewed from the side.
  • the horizontal and vertical directions of the image (captured image) in the observation range captured by the camera 21 are defined as the x direction and the y direction, respectively.
  • the coordinate system set based on the position of the camera 21 is referred to as a camera coordinate system.
  • the camera coordinate system is, for example, a coordinate system represented by two-dimensional coordinates (x, y) and the z direction orthogonal to the two-dimensional coordinates, and the position of the camera 21 is set as the origin.
  • the horizontal direction (horizontal direction) of the display panel 20 is the X direction
  • the vertical direction of the display panel 20 is the Y direction.
  • the direction orthogonal to the surface (XY plane) of the display panel 20 is defined as the Z direction.
  • the coordinate system represented by the X direction, the Y direction, and the Z direction is described as a display coordinate system (X, Y, Z). Further, as shown in FIG.
  • the rotation angle with the X direction as the rotation axis is defined as ⁇
  • the rotation angle with the Y direction as the rotation axis is ⁇
  • the rotation angle with the Z direction as the rotation axis is described as ⁇ . Therefore, the position and orientation in the display coordinate system are represented by (X, Y, Z, ⁇ , ⁇ , ⁇ ).
  • the display coordinate system will be used as the world coordinates in the stereoscopic display device 100. For example, when displaying the stereoscopic image 10 (target object 14 or virtual pointer 15), it is assumed that the position and orientation in the display coordinate system are calculated.
  • the position of the marker 41 (pointing device 40) in the display coordinate system is described as M (X M , Y M , Z M ), and the axial direction of the pointing device 40 and the surface (XY surface) of the display panel 20 are defined.
  • the intersection is described as D (X D , Y D , 0), and the contact point between the virtual pointer 15 and the target object 14 is described as O (X obj , Y obj , Z obj ).
  • the intersection D is the contact position between the display panel 20 and the pointing device 40.
  • 5A to 5C show the marker position M, the intersection D, and the contact point O, respectively.
  • the pointer detection unit 32 first detects the marker 41 from the captured image captured by the camera 21 (step 101). Image recognition processing using, for example, feature point detection, template matching, or the like is used to detect the marker 41.
  • the method for detecting the marker 41 is not limited, and the marker 41 may be detected by using, for example, machine learning.
  • the pointer detection unit 32 determines whether or not the marker 41 has been detected (step 102). If the marker 41 is not detected (No in step 102), step 101 is executed again, and the detection process of the marker 41 is executed for the next image. When the marker 41 is detected (Yes in step 102), the pointer detection unit 32 executes a process of estimating the position and orientation of the marker 41 (step 103).
  • the position and orientation of the marker 41 is used as the position and orientation of the pointing device 40. That is, the pointer detection unit 32 detects the position / orientation of the marker 41 as the position / orientation of the pointing device 40. By using the marker 41, it is possible to estimate the position and orientation of the pointing device 40 with high accuracy. The position and orientation of the pointing device 40 may be calculated by appropriately converting the position and orientation of the marker 41.
  • the two-dimensional coordinates (x i , y i ) of each detection point 47 of the marker 41 in the captured image are calculated.
  • i is an index representing each detection point 47.
  • These two-dimensional coordinates are coordinates in which the three-dimensional coordinates of the detection point 47 in the real space are projected (transparent projection conversion) in the captured image by being imaged by the camera 21.
  • the internal parameter is a parameter that changes according to zoom and the like, and includes, for example, a parameter that represents the principal point (typically the center of the image) of the captured image and a parameter that represents the focal length corresponding to each pixel.
  • the external parameter is a parameter that changes according to the position and orientation of the camera 21, and includes, for example, a matrix element of a simultaneous transformation matrix of translation and rotation in fluoroscopic projection transformation. These parameter values are pre-calibrated with reference to, for example, the display coordinate system and stored in the storage unit 26.
  • the translation and rotation vectors of the marker 41 appearing in the two-dimensional image (captured image) are derived by using the solution of the PNP problem.
  • the two-dimensional coordinates (x i, y i) of each detection point 47 when converted into three-dimensional coordinates, in the actual marker 41 The translation vector and the rotation vector of the marker 41 are calculated so as to satisfy the three-dimensional arrangement relationship of each detection point 47.
  • vectors (x c , y c , z c , ⁇ c , ⁇ c , ⁇ c ) representing the position and orientation of the marker 41 in the camera coordinate system are calculated.
  • the pointer detection unit 32 converts the position and orientation of the marker 41 in the camera coordinate system calculated in step 103 into the position and orientation of the display coordinate system (step 104).
  • a transformation matrix set according to the arrangement relationship between the camera 21 and the display panel 20 is used for the transformation of the coordinate system.
  • the vector representing the position and orientation of the marker 41 is converted into a vector (X M , Y M , Z M , ⁇ M , ⁇ M , ⁇ M ) in the display coordinate system by the transformation matrix.
  • the vector representing the position and orientation of the marker 41 in the display coordinate system is the position and orientation of the pointing device 40.
  • the pointer detection unit 32 detects the position and orientation of the pointing device 40 in the display coordinate system, that is, the position and orientation of the pointing device 40 with respect to the display panel 20.
  • FIG. 6 is a schematic view showing an example of a method of calculating the position and orientation of the pointing device 40.
  • FIG. 6 shows the position M of the marker 41 (pointing device 40) in the display coordinate system (X, Y, Z) with the intersection D between the axial direction of the pointing device 40 and the display panel 20 as the origin. ..
  • the pointer detection unit 32 calculates the intersection D between the pointing device 40 and the display panel 20 (step 105).
  • the position M (X M , Y M , Z M ) of the marker 41 and the rotation angle ⁇ ( ⁇ M , ⁇ M , ⁇ M ) of the marker 41 in the display coordinate system are calculated in step 104 and are known values. is there.
  • the X and Y coordinates of the intersection D are set as X D and Y D
  • the Z coordinate (Z M ) of the marker 41 is represented by using the position M and the rotation angle ⁇ of the marker 41.
  • the Z coordinate of the marker 41 is represented by the following equation using, for example, the length (X M ⁇ X D ) in the X direction from the intersection D to the position M of the marker 41.
  • the Z coordinate of the marker 41 is expressed by the following equation using, for example, the length in the Y direction (Y M ⁇ Y D ) from the intersection D to the position M of the marker 41.
  • the Z coordinate (Z M ) of the marker 41 in the display coordinate system is described by two methods as the height seen from the XY plane.
  • the left side of the equation (4) includes the X coordinate (X D ) of the intersection D to be calculated. Therefore, by modifying equation (4) for X D , the following equation is obtained.
  • the X coordinate (X D ) and Y coordinate (Y D ) of the intersection D are the positions M (X M , Y M , Z) of the marker M that have already been calculated. M), and the rotation angle of the marker 41 ( ⁇ M, can be expressed using phi M). That is, the intersection D between the axial direction of the pointing device 40 and the surface of the display panel 20 is represented as follows.
  • the pointer detection unit 32 calculates the X coordinate and the Y coordinate of the intersection D in the display coordinate system according to the equation (7).
  • the intersection D may be calculated by using another method without being limited to the above method.
  • the display control unit 33 calculates the length of the pointing device 40 (step 106).
  • the length of the pointing device 40 is, for example, the length from the position of the tip 46 to the position M of the marker 41.
  • the position of the tip 46 changes due to expansion and contraction.
  • the length from the tip 46 to the marker 41 in the non-contact state in which the pointing device 40 is not shrunk is described as the reference length d.
  • the length of the pointing device 40 is the reference length d.
  • the telescopic portion 42 expands and contracts, so that the length of the pointing device 40 is equal to or less than the reference length d.
  • the length of the current pointing device 40 is calculated based on the distance MD from the intersection D to the marker position M.
  • the distance MD is expressed by the following equation using the coordinates of the intersection D calculated by the equation (7) and the position M of the marker 41.
  • the length of the pointing device 40 becomes the reference length.
  • the length of the pointing device 40 is the distance MD.
  • step 107 it is determined whether or not the pointing device 40 and the display panel 20 are in contact with each other (step 107). Specifically, using the reference length d as a threshold value, it is determined whether or not the distance MD is equal to or less than the reference length d. This is a determination as to whether or not the current length of the pointing device 40 is shorter than the length of the pointing device 40 in the non-contact state (the original length of the pointing device 40). When the distance MD is longer than the reference length d (MD> d), it is determined that the pointing device 40 is not in contact with the display panel 20 (No in step 107). In this case, the processes after step 101 are executed for the next image.
  • the reference length d as a threshold value
  • the process of drawing the virtual pointer 15 is executed according to the length (distance MD) of the pointing device 40 (step 108).
  • FIG. 7 is a schematic diagram showing a display example of the virtual pointer 15. On the left and right sides of FIG. 7, a pointing device 40 in a non-contact state and a pointing device 40 that has shrunk in contact with the display panel 20 are schematically illustrated.
  • the display control unit 33 calculates a virtual insertion amount into the display panel 20 (display space 11) of the pointing device 40, which is the length of the virtual pointer 15.
  • the expansion / contraction amount ⁇ of the pointing device 40 in response to the contact between the display panel 20 and the pointing device 40 is calculated as a virtual insertion amount of the pointing device 40 into the display space 11.
  • the pointing device 40 shrinks by the amount that the tip 46 is pressed against the display panel 20.
  • the expansion / contraction amount ⁇ at this time is calculated as a virtual insertion amount of the pointing device 40 into the display panel 20 (display space 11).
  • the display control unit 33 generates a virtual pointer 15 based on the expansion / contraction amount ⁇ , the posture of the pointing device 40, and the contact position (intersection point D) with the display panel 20.
  • the virtual pointer 15 is generated based on, for example, the design data stored in the object library 28.
  • the display control unit 33 displays the virtual pointer 15 starting from the contact position (intersection D) between the display panel 20 and the pointing device 40 in the display space 11. For example, as shown on the right side of FIGS. 1B and 7, the virtual pointer 15 is displayed so as to extend from the intersection D where the pointing device 40 contacts the display panel 20 to the display space 11.
  • the display control unit 33 displays the virtual pointer 15 along the axial direction of the pointing device 40. That is, the virtual pointer 15 that extends and displays the pointing device 40 in the direction in which the pointing device 40 is directed is displayed in the display space 11. As a result, it is possible to realize a display as if the pointing device 40 is inserted into a virtual display space, and an intuitive input operation is possible.
  • the display of the virtual pointer 15 is controlled so that the length of the virtual pointer 15 is proportional to the expansion / contraction amount ⁇ .
  • a virtual pointer 15 having the same length as the expansion / contraction amount ⁇ is displayed in the display space 11.
  • the proportional coefficient between the length of the virtual pointer 15 and the expansion / contraction amount ⁇ is 1, and the offset or the like is not set.
  • a virtual pointer 15 having a length obtained by adding a predetermined offset to the expansion / contraction amount ⁇ may be displayed.
  • the virtual pointer 15 is an object longer (or shorter) than the expansion / contraction amount ⁇ .
  • the length of the virtual pointer 15 is set in proportion to the expansion / contraction amount ⁇ .
  • the proportional coefficient between the length of the virtual pointer 15 and the expansion / contraction amount ⁇ may be appropriately set.
  • a proportional coefficient of 1 or more may be set.
  • a proportional coefficient smaller than 1 may be set.
  • the operation amount of the user 1 can be reduced, and for example, fine input operations can be easily performed.
  • the proportional coefficient and the offset it is possible to display the virtual pointer 15 that is linearly linked to the operation of the user 1, and it is possible to perform an intuitive input operation.
  • the display control unit 33 draws the virtual pointer 15 according to the difference (expansion / contraction amount ⁇ ).
  • step 109 it is determined whether or not the virtual pointer 15 and the target object 14 are in contact with each other. For example, whether or not a part (tip, side surface, etc.) of the virtual pointer 15 is in contact with the target object 14 is determined based on the coordinate values of the display coordinates.
  • step 108 is executed and the drawing of the virtual pointer 15 is continued.
  • the processing after step 101 may be executed again.
  • the display control unit 33 executes a process of changing the drawing of the target object 14 (step 110).
  • the process of changing the drawing of the target object 14 is a process of displaying the reaction (interaction) of the operation of the virtual pointer 15 with respect to the target object 14.
  • This process is executed according to, for example, the contact position and the contact amount with the virtual pointer 15.
  • the contact amount is, for example, an amount representing the degree to which the virtual pointer 15 interferes with the target object 14, and is expressed using, for example, the amount of penetration of the virtual pointer 15 into the coordinate space inside the target object 14.
  • the type of interaction with the target object 14 is not limited. For example, there is a process of deforming the target object 14 itself, such as denting the contact position with which the virtual pointer 15 is in contact according to the contact amount of the virtual pointer 15.
  • a process of changing the position of the target object 14, such as moving the target object 14 or changing the direction of the target object 14, may be executed in response to the contact of the virtual pointer 15.
  • the movement amount and the rotation amount of the target object 14 are appropriately calculated according to the contact position, the contact amount, and the like.
  • a process of drawing the target object 14 so as to make a predetermined reaction according to the contact may be executed.
  • the process of looking back at the rabbit object described with reference to FIG. 1 is a process of drawing a reaction in response to contact with the virtual pointer 15.
  • the display control unit 33 controls the display of the target object 14 according to the contact between the target object 14 and the virtual pointer 15. This makes it possible to present various interactions according to input operations (touching, poking, hitting, cutting, pushing, etc.) with respect to the target object 14.
  • the vibration control unit 34 executes a feedback process for presenting vibration to the user 1 (step 111). For example, the vibration control unit 34 generates a vibration signal according to the content and the like, and outputs the vibration signal to the vibration presenting unit 45 mounted on the pointing device 40. As a result, the pointing device 40 generates vibration in response to the vibration signal, and the user 1 is presented with a tactile sensation.
  • the vibration control unit 34 controls the vibration presentation unit 45 according to at least one of the type of the target object 14, the type of the virtual pointer 15, or the contact amount between the target object 14 and the virtual pointer 15. .
  • vibration signals that present various vibration sensations are generated according to the types of the target object 14 and the virtual pointer 15. Further, the intensity of these vibration sensations is adjusted according to the contact amount.
  • the method of controlling the vibration presenting unit 45, the method of generating a vibration signal, and the like are not limited.
  • the skin sensation is fed back to the user 1 through the pointing device 40. As a result, it is possible to present a feeling of being in contact with an existing object, and it is possible to improve the operability of the input operation.
  • FIGS. 1B and 7 described above a rod-shaped virtual pointer 15 that is simply an extension of the pointing device 40 has been described as an example.
  • the shape of the virtual pointer 15 and the like are not limited, and any design can be applied according to the use case.
  • a knife-shaped object may be used as the virtual pointer 15. In this case, it is possible to perform an operation of disconnecting the target object 14 using the virtual pointer 15. Further, an object having a pointed tip may be used. In this case, it is possible to perform an operation of stabbing the target object 14.
  • an object having a tip shaped like a brush or a brush may be used. In this case, it is possible to paint the target object 14 or to draw a picture or a character. Further, an object having a tip in the shape of a tool such as a screwdriver or a spanner may be used. In this case, the operation of assembling or disassembling the target object 14 becomes possible. Further, by using a plurality of pointing devices at the same time, it may be possible to perform an operation such as pinching the target object 14 with chopsticks. Also, by using a plurality of devices, it is possible to use a plurality of tools at the same time in the display space. Further, for example, the type of the virtual pointer 15 may be set according to the type of the target object 14.
  • the virtual pointer 15 in the shape of a knife, fork, or the like is automatically displayed.
  • the pointing device 40 is used to input to the human body or the affected part (target object 14) displayed in three dimensions.
  • the virtual pointer 15 displays an object of a medical instrument such as a scalpel or forceps.
  • a predetermined vibration is presented via the device. This makes it possible to simulate surgery.
  • Other examples of use include architecture, model design, and work. In this case, a model, a part (target object 14), or the like is three-dimensionally displayed, and a virtual pointer 15 that imitates a tool or the like for precision machining is used.
  • the pointing device 40 can be used for intuitive operation, it is possible to perform a complicated input operation on the object to be processed or assembled.
  • the input interface using the pointing device 40 and the virtual pointer 15 can also be used as a communication tool. For example, it is possible to perform interactive communication with a person in a remote place by using a plurality of stereoscopic display devices 100. In this case, for example, it is possible to intuitively operate each other on a shared object displayed in three dimensions.
  • a target object 14 virtual pet
  • a target object 14 virtual pet
  • the user 1 can come into contact with the virtual pet virtually displayed through the pointing device 40 and experience the breeding of the pet.
  • the scope of application of this technology is not limited to the above-mentioned examples, and can be applied to various scenes such as amusement, education, and product development.
  • the position and orientation of the pointing device 40 are detected from the image of the pointing device 40 used by the user 1. Based on this detection result, the virtual insertion portion of the pointing device 40 into the display space 11 of the stereoscopic display device 100 is displayed as a virtual pointer 15. As a result, the virtual pointer 15 in the display space 11 can be easily operated by using the pointing device 40, and an intuitive input operation for the stereoscopic image 10 can be realized.
  • the pointing device 40 by operating the pointing device 40 so as to bring it into direct contact with the target object 14, it is possible to perform an input operation on the target object 14 via the virtual pointer 15. In this way, it is possible to operate a virtual object as it is using a real device, and it is possible to demonstrate high usability.
  • the virtual pointer 15 comes into contact with the target object 14, sensory feedback is performed by the vibration presenting unit 45 or the like. This makes it possible to provide a virtual experience as if operating a real object.
  • a method of performing an input operation in the depth direction there is a method of detecting the position and orientation of an input device using a sensor.
  • a method of calculating the position and orientation of the input device by using a touch sensor mounted on the display, a posture sensor mounted on the input device, or the like can be considered. In such a method, it is necessary to use a special sensor, which may lead to an increase in equipment cost.
  • the image of the pointing device 40 is captured by using the camera 21 provided in the stereoscopic display device 100. From this image, the position and orientation of the pointing device 40 are calculated, and the display control of the virtual pointer 15 and the like is performed. As described above, in the present embodiment, the display processing of the virtual pointer 15 can be performed without providing a touch sensor, a position / orientation sensor, or the like. As a result, the cost of the device can be sufficiently suppressed. In particular, in a type of device such as the stereoscopic display device 100 described above that displays a stereoscopic image according to the viewpoint 2 of the user 1, a camera 21 for tracking the viewpoint 2 is often provided. By using such a camera 21, it is possible to easily implement the display processing of the virtual pointer 15.
  • the virtual pointer is displayed so that its length is proportional to the amount of expansion and contraction of the pointing device (the amount of virtual insertion into the display panel).
  • the virtual pointer may be controlled non-linearly with respect to the amount of expansion and contraction. For example, control may be performed such that the larger the expansion / contraction amount, that is, the larger the amount of pushing the pointing device, the larger the rate of increasing the length of the virtual pointer.
  • the virtual pointer may be displayed along a direction different from the axial direction of the pointing device. This makes it possible to perform complicated input operations in a comfortable posture.
  • a stereoscopic display device that performs stereoscopic display with the naked eye has been described.
  • a glasses-type stereoscopic display device that displays a stereoscopic image using a glasses-type device such as polarized glasses or liquid crystal shutter glasses may be used.
  • the user wears a glasses-type device for perceiving a parallax image and observes the display panel. Even in this case, it is possible to display the virtual pointer according to the amount of expansion and contraction of the pointing device.
  • the present technology can also be applied when stereoscopic display is performed using a head-mounted display (HMD) or the like.
  • the pointing device does not need to expand or contract.
  • HMD head-mounted display
  • the pointing device does not need to expand or contract.
  • an HMD capable of AR display it is possible to display a stereoscopic image (right eye image and left eye image) by superimposing it on the field of view in front of the user.
  • the range of the field of view in which the stereoscopic image is displayed is the display space of the HMD.
  • the image of the pointing device is captured by using a front camera or the like mounted on the HMD. Suppose the user brings the tip of the pointing device closer to the target object.
  • the virtual pointer is displayed by superimposing it on the pointing device that has entered the user's field of view (display space). This enables intuitive input operations via the virtual pointer. Further, for example, when a boundary surface is set in the display space and the pointing device exceeds the boundary surface, a virtual pointer having a length corresponding to the insertion amount is displayed. For example, such a process may be executed.
  • the type of device that performs stereoscopic display is not limited.
  • this technology can also adopt the following configurations.
  • a detection unit that detects the position and orientation of the indicator based on the image of the indicator used by the user, and An information processing device including a display control unit that displays an instruction object representing a virtual insertion portion of the indicator in the display space in the display space of the stereoscopic display device based on the position and orientation of the indicator.
  • the information processing device according to (1) The display control unit calculates a virtual insertion amount of the indicator into the display space based on the position and orientation of the indicator, and controls the display of the instruction object according to the insertion amount. .. (3) The information processing device according to (2).
  • the display control unit is an information processing device that controls the display of the instruction object so that the length of the instruction object is proportional to the insertion amount.
  • the stereoscopic display device has a display panel for displaying a stereoscopic image.
  • the display space is a virtual space with the display panel as a boundary.
  • the detection unit is an information processing device that detects the position and orientation of the indicator with respect to the display panel.
  • the indicator can be expanded and contracted in one direction in response to pressure.
  • the display control unit is an information processing device that calculates the amount of expansion and contraction of the indicator in response to contact between the display panel and the indicator as a virtual insertion amount of the indicator into the display space.
  • the information processing device is an information processing device that displays the instruction object starting from the contact position between the display panel and the indicator in the display space.
  • the information processing apparatus according to any one of (4) to (6).
  • the stereoscopic display device has an image pickup unit directed to an observation range of the display panel.
  • the detection unit is an information processing device that detects the position and orientation of the indicator used by the user in the observation range based on the image of the observation range captured by the imaging unit.
  • the information processing apparatus according to (7).
  • the detection unit detects the viewpoint of the user who observes the display panel based on the image of the observation range captured by the imaging unit.
  • the display control unit is an information processing device that displays the stereoscopic image according to the viewpoint of the user.
  • the information processing apparatus has a marker portion and has a marker portion.
  • the detection unit is an information processing device that detects the position and orientation of the marker unit as the position and orientation of the indicator.
  • the indicator has a tip that is directed at the object and a grip that is connected to the tip.
  • the marker portion is an information processing device connected to a side opposite to the side to which the tip portion of the grip portion is connected.
  • the display control unit is an information processing device that displays a target object that is a target of an input operation using the indicator in the display space. (12) The information processing apparatus according to (11).
  • the display control unit is an information processing device that sets the type of the instruction object according to the type of the target object. (13) The information processing device according to (11) or (12). The display control unit is an information processing device that controls the display of the target object in response to contact between the target object and the instruction object. (14) The information processing apparatus according to any one of (11) to (13).
  • the indicator has a sensation presenting portion that presents a skin sensation to the user. Further, an information processing device including a sensory control unit that controls the sensory presentation unit in response to contact between the target object and the instruction object. (15) The information processing apparatus according to (14).
  • the sensation presenting unit is an information processing device that presents at least one of a vibration sensation and a heat sensation. (16) The information processing apparatus according to (14) or (15).
  • the sensory control unit is an information processing device that controls the sensory presentation unit according to at least one of the type of the target object, the type of the instruction object, or the amount of contact between the target object and the instruction object.
  • the position and orientation of the indicator are detected based on the image of the indicator used by the user.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Optics & Photonics (AREA)
  • Position Input By Displaying (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Pour atteindre le but de la présente invention, un dispositif de traitement d'informations selon un mode de réalisation comprend une unité de détection et une unité de commande d'affichage. L'unité de détection détecte, sur la base d'une image d'un outil d'indication utilisé par un utilisateur, la position/l'orientation de l'outil d'indication. L'unité de commande d'affichage affiche, dans un espace d'affichage d'un dispositif d'affichage stéréoscopique et sur la base de la position/de l'orientation de l'outil d'indication, un objet d'indication qui représente une partie d'insertion virtuelle de l'outil d'indication par rapport à l'espace d'affichage.
PCT/JP2020/029699 2019-08-13 2020-08-03 Dispositif de traitement d'informations, procédé de traitement d'informations et programme Ceased WO2021029256A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019148558 2019-08-13
JP2019-148558 2019-08-13

Publications (1)

Publication Number Publication Date
WO2021029256A1 true WO2021029256A1 (fr) 2021-02-18

Family

ID=74569567

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/029699 Ceased WO2021029256A1 (fr) 2019-08-13 2020-08-03 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Country Status (1)

Country Link
WO (1) WO2021029256A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022230350A1 (fr) * 2021-04-28 2022-11-03 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme
WO2023195301A1 (fr) * 2022-04-04 2023-10-12 ソニーグループ株式会社 Dispositif de commande d'affichage, procédé de commande d'affichage et programme de commande d'affichage
JP7777842B1 (ja) * 2025-06-14 2025-12-01 インターマン株式会社 空中に表示された立体オブジェクトとの接触を可視化するプログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003085590A (ja) * 2001-09-13 2003-03-20 Nippon Telegr & Teleph Corp <Ntt> ３次元情報操作方法およびその装置，３次元情報操作プログラムならびにそのプログラムの記録媒体
JP2011087848A (ja) * 2009-10-26 2011-05-06 Mega Chips Corp ゲーム装置
JP2013097805A (ja) * 2011-11-03 2013-05-20 Optoelectronics Co Ltd Shenzhen Super Multi 立体的インタラクティブシステム及び立体的インタラクティブ方法
JP2018018309A (ja) * 2016-07-28 2018-02-01 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003085590A (ja) * 2001-09-13 2003-03-20 Nippon Telegr & Teleph Corp <Ntt> ３次元情報操作方法およびその装置，３次元情報操作プログラムならびにそのプログラムの記録媒体
JP2011087848A (ja) * 2009-10-26 2011-05-06 Mega Chips Corp ゲーム装置
JP2013097805A (ja) * 2011-11-03 2013-05-20 Optoelectronics Co Ltd Shenzhen Super Multi 立体的インタラクティブシステム及び立体的インタラクティブ方法
JP2018018309A (ja) * 2016-07-28 2018-02-01 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022230350A1 (fr) * 2021-04-28 2022-11-03 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, et programme
WO2023195301A1 (fr) * 2022-04-04 2023-10-12 ソニーグループ株式会社 Dispositif de commande d'affichage, procédé de commande d'affichage et programme de commande d'affichage
JP7777842B1 (ja) * 2025-06-14 2025-12-01 インターマン株式会社 空中に表示された立体オブジェクトとの接触を可視化するプログラム

Similar Documents

Publication Publication Date Title
CN103443742B (zh) 用于凝视和姿势接口的系统和方法
EP2755194B1 (fr) Système d&#39;entraînement virtuel 3d et procédé
US10007351B2 (en) Three-dimensional user interface device and three-dimensional operation processing method
CN103180893B (zh) 用于提供三维用户界面的方法和系统
EP3311249B1 (fr) Entrée de données tridimensionnelles d&#39;utilisateur
KR101171660B1 (ko) 증강현실의 포인팅 장치
CN103744518B (zh) 立体交互方法及其显示装置和系统
CN106454311B (zh) 一种led三维成像系统及方法
KR101666561B1 (ko) 증강 공간 내 부분 공간 획득 시스템 및 방법
CN114647317B (zh) 由外围设备启用的远程触摸检测
KR101518727B1 (ko) 입체 인터랙션 시스템 및 입체 인터랙션 방법
CN101995943B (zh) 立体影像互动系统
KR20140010616A (ko) 3ｄ 가상 객체에 대한 조작 처리 장치 및 방법
JP2009276996A (ja) 情報処理装置、情報処理方法
CN111273766B (zh) 用于生成链接到物品模拟现实表示的示能表示的方法、设备和系统
JP6344530B2 (ja) 入力装置、入力方法、及びプログラム
WO2021029256A1 (fr) Dispositif de traitement d&#39;informations, procédé de traitement d&#39;informations et programme
CN109952552A (zh) 视觉提示系统
TW202336562A (zh) 控制和虛擬物件的互動
CN104285243A (zh) 三维图像显示设备及其光标显示方法和计算机程序
Gigante Virtual reality: Enabling technologies
US11403830B2 (en) Image processing device, image processing method, and program
JP5863984B2 (ja) ユーザインタフェース装置及びユーザインタフェース方法
Aspin et al. Augmenting the CAVE: An initial study into close focused, inward looking, exploration in IPT systems
US11086505B2 (en) Program, information processing device, and control method for the same

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20851696

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20851696

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP