WO2024218977A1 - Aerial image display system, display control device, method, and program - Google Patents

Abstract

An embodiment of the present invention comprises: a display control device that causes a display device to display a display video; an aerial image optical system that displays, as an aerial image and in a display space, an optical image from the display video displayed on the display device; and a position sensor that detects position information indicating the position of a fingertip of a user in the display space when the user performs a pointing operation with respect to the aerial image. The display control device is configured to perform, on the basis of the position information detected by the position sensor and display position information of the aerial image in the display space, first processing to generate auxiliary information representing a positional relationship between the fingertip of the user and the aerial image, and second processing to generate a display video equipped with auxiliary information obtained by superimposing the auxiliary information on the display video and to cause the display device to display the generated display video equipped with the auxiliary information.

Description

Aerial image display system, display control device, method and program

One aspect of the present invention relates to an aerial image display system for displaying, for example, an image displayed on a display in space, and a display control device, method, and program used in the system.

Various aerial image display technologies have been proposed in the past that give the perception that an image is floating in the air. For example, Non-Patent Document 1 describes a technology related to AIRR, which displays an image displayed on a display as an aerial image using an aerial image optical system that uses a retroreflective material. Non-Patent Document 2 describes a technology that uses a retrotransparent material (MMAP) in the aerial image optical system. All of these technologies display an aerial image by forming the image displayed on a display, which acts as a light source, as a real image in real space.

Moreover, recently, a pointing technology has been proposed that allows a user to select an object displayed as an aerial image as described above, thereby executing a specific process associated with the object. Using this technology, a user can perform a desired operation without directly touching an operation panel, for example.

Hirotsugu Yamamoto, "Aerial Display Using Aerial Imaging by Retroreflection (AIRR)," Journal of the Imaging Society of Japan, Vol. 56, No. 4, pp. 341-351 (2017) Mayumi Takasaki, Shinji Mizuno, "Proposal of a method to reproduce occlusion of real objects by aerial CG objects in an aerial 3D imaging system", Information Processing Society of Japan Interaction 2020, 2C-61

However, the aerial image formed in space has a blurred image from the display, which acts as a light source, and so the contours become unclear. This makes it difficult for the accommodation function, which is one aspect of human depth perception, to function when the user points to the aerial image. Furthermore, when the user places his or her fingertip over the aerial image to perform a pointing operation, if the finger is positioned deeper than the aerial image, an occlusion contradiction arises in the overlapping relationship between the aerial image and the finger. For these reasons, the inhibition of accommodation and the influence of the occlusion contradiction reduce the user's depth perception, and pointing accuracy decreases, particularly when pointing by placing the fingertip over the aerial image, including the depth position.

This invention was made with the above in mind, and aims to provide a technology that enables high pointing accuracy when pointing to an aerial image where depth perception is reduced.

In order to solve the above problems, one aspect of the aerial image display system according to the present invention includes a display control device that displays a display image on a display device, an aerial image optical system that displays an optical image based on the display image displayed on the display device as an aerial image in a display space, and a position sensor that detects position information that represents the position of the user's fingertip in the display space when the user performs a pointing operation on the aerial image. The display control device then performs a first process of generating auxiliary information that represents the positional relationship between the user's fingertip and the aerial image based on the position information detected by the position sensor and display position information of the aerial image in the display space, and a second processing unit that generates a display image with auxiliary information by superimposing the auxiliary information on the display image and displays the generated display image with auxiliary information on the display device.

According to one aspect of the present invention, auxiliary information is generated that represents the positional relationship between the user's fingertip and the aerial image when performing a pointing operation on the aerial image, and this auxiliary information is displayed in the display space in a state where it is associated with the aerial image.

As a result, the user can refer to the auxiliary information to grasp the depth position of the aerial image without being affected by occlusion from the fingers, thereby reducing the effects of reduced depth perception due to occlusion inconsistencies and enabling highly accurate pointing operations. In addition, the user can refer to the displayed auxiliary information to correct misalignment of his or her fingertip relative to the aerial image, thereby particularly reducing the effects of reduced depth perception and enabling highly accurate pointing operations.

In other words, one aspect of the present invention provides a technology that enables high pointing accuracy to be achieved when performing a pointing operation on an aerial image where depth perception is reduced.

FIG. 1 is a schematic diagram of an aerial image display system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of an aerial image optical system in the aerial image display system shown in FIG. FIG. 3 is a block diagram showing an example of a hardware configuration of a display control device in the aerial image display system shown in FIG. FIG. 4 is a block diagram showing an example of a software configuration of the display control device in the aerial image display system shown in FIG. FIG. 5 is a flowchart showing an example of a processing procedure and processing content of the aerial image display control executed by the display control device shown in FIG. FIG. 6 is a diagram showing a first example of a display operation of auxiliary information that assists a user in perceiving depth. FIG. 7 is a diagram showing a second example of the display operation of auxiliary information for assisting depth perception.

Below, an embodiment of the present invention will be described with reference to the drawings.

[One embodiment]
(Configuration example)
(1) Aerial Image Display System FIG. 1 is a schematic diagram of an aerial image display system according to an embodiment of the present invention.

The aerial image display system of one embodiment includes an aerial image optical system DU that displays an aerial image, a three-dimensional position sensor SU, and a display control device CU. Note that HD in the figure indicates the user's hand, and the user performs a pointing operation on the aerial image using the fingers of the hand HD.

The aerial image optical system DU, as shown in FIG. 2, for example, comprises a display DP as a light source, a beam splitter HM, and a retroreflective member RM. The retroreflective member RM is positioned so that its reflective surface is perpendicular or nearly perpendicular to the display surface of the display DP. The beam splitter HM is positioned between the user's viewpoint and the retroreflective member RM, with its active surface at a predetermined angle, for example 45 degrees or nearly so, to the user's viewpoint. The beam splitter RM has the optical property of transmitting part of the incident light and reflecting part of it, and is composed of, for example, a half mirror.

The three-dimensional position sensor SU detects the three-dimensional position coordinates in space of the fingertip of the user performing a pointing operation, and is configured, for example, with a motion camera such as a Kinect (registered trademark) camera. Note that any sensor device capable of detecting the three-dimensional position coordinates of the fingertip may be used as the three-dimensional position sensor SU.

(2) Display control unit CU
2 and 3 are block diagrams showing examples of hardware and software configurations of a display control unit CU used in an aerial image display system according to an embodiment of the present invention.

The display control device CU is, for example, a personal computer, and has a control unit 1 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, and an input/output interface (hereinafter the interface will be abbreviated as I/F) unit 4 are connected to this control unit 1 via a bus 5.

The display DP and three-dimensional position sensor SU are connected to the input/output I/F unit 4. The display DP is a flat display device using, for example, liquid crystal or organic electroluminescence. Note that other input devices such as a keyboard and external storage media may also be connected to the input/output I/F unit 4. The input/output I/F unit 4 may also be provided with a communication interface that employs a low-power data wireless communication standard such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). By connecting the display DP and three-dimensional position sensor SU to the display control device CU using a wireless interface, the freedom of positioning of the display control device CU can be increased.

The program storage unit 2 is configured by combining, for example, a non-volatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for control according to one embodiment of the present invention, in addition to middleware such as an operating system (OS). Note that hereafter, the OS and each application program will be collectively referred to as the program.

The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium, and in the storage area, an auxiliary pattern storage unit 31 and a display content storage unit 32 are provided as storage units used in one embodiment of the present invention.

The display content storage unit 32 is used to store the video data of the display content that is the basis for the aerial image. For example, the display content may be an operation panel or a display image of various switches, but other display images of context information such as pictures, charts, news, etc. may also be used.

The auxiliary pattern storage unit 31 stores an auxiliary pattern that represents the positional relationship in the depth direction between the user's fingertip and the aerial image. An example of the auxiliary pattern is shown in the operation example.

The control unit 1 includes, as processing functions necessary to implement one embodiment of the present invention, a fingertip position information acquisition processing unit 11, an auxiliary information generation processing unit 12, an aerial image display processing unit 13, and a display content acquisition processing unit 14. All of these processing units 11 to 14 are realized by causing the hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2.

In addition, some or all of the above processing units 11 to 14 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

The display content acquisition processing unit 14 acquires display content, for example, from a website, and stores the acquired display content in the display content storage unit 32. Note that the display content may be arbitrarily created by a system administrator or the user himself, and the type of data may be, for example, any of photographs, diagrams, CG data, and text data.

The fingertip position information acquisition processing unit 11 acquires coordinate data representing the three-dimensional position of the user's fingertip from the three-dimensional position sensor SU via the input/output I/F unit 4 at a constant period set in advance.

The auxiliary information generation processing unit 12 calculates the distance in the depth direction between the aerial image and the user's fingertip from the display position coordinates in the display space of the aerial image displayed by the aerial image display processing unit 13 (described later) and the three-dimensional position coordinates of the user's fingertip acquired by the fingertip position information acquisition processing unit 11. The auxiliary information generation processing unit 12 then generates auxiliary information by reflecting the distance in the auxiliary pattern read out from the auxiliary pattern storage unit 31, and provides the generated auxiliary information to the aerial image display processing unit 13.

The aerial image display processing unit 13 generates display video data by superimposing the auxiliary information on the video data of the display content read from the display content storage unit 32, and outputs the generated display video data from the input/output I/F unit 4 to the display DP for display.

(Example of operation)
Next, an example of an aerial image display operation by the aerial image display system configured as above will be described.

FIG. 5 is a flowchart showing an example of the processing procedure and processing content of the aerial image display control executed by the control unit 1 of the display control device CU.

(1) Display of Aerial Image For example, assume that an aerial image display start request is input from the user terminal in response to a user operation, and the control unit 1 of the display control device CU detects this input of the aerial image display start request in step S10.

Then, in step S11, the control unit 1 of the display control device CU first acquires display content, for example, from a website via the input/output I/F unit 4 under the control of the display content acquisition processing unit 14, and stores the acquired display content in the display content storage unit 32. Note that the display content may be created by a system administrator or a user, as described above. The display content may also be stored in advance in the display content storage unit 32.

Next, in step S12, the control unit 1 of the display control device CU, under the control of the aerial image display processing unit 13, reads the video data of the display content from the display content storage unit 32, and outputs the read video data of the display content from the input/output I/F unit 4 to the display DP.

As a result, the video data of the display content is displayed on the display screen of the display DP. The optical image of the displayed content is then reflected by the beam splitter HM, then retroreflected by the retroreflecting member RM, and then transmitted through the beam splitter HM to be displayed as an aerial image RI consisting of a real image in the display space.

(2) Acquisition of User's Fingertip Position Coordinates When the aerial image RI of the display content is displayed, it is assumed that the user brings his/her finger close to the aerial image RI in order to perform a pointing operation on the aerial image RI. In this case, the three-dimensional position coordinates of the user's fingertip in the display space are detected by the three-dimensional position sensor SU.

In response to this, in step S13, the control unit 1 of the display control device CU, under the control of the fingertip position information acquisition processing unit 11, acquires information representing the three-dimensional position coordinates of the fingertip detected by the three-dimensional position sensor SU at regular time intervals via the input/output I/F unit 4. Then, the acquired three-dimensional position coordinate information of the fingertip is passed to the auxiliary information generation processing unit 12. In other words, the fingertip position information acquisition processing unit 11 obtains position information indicating changes in the three-dimensional position of the user's fingertip in the display space.

If, for example, a depth camera is used as the three-dimensional position sensor SU, the fingertip position information acquisition processing unit 11 may calculate the three-dimensional position coordinates of the fingertip based on the image data of the fingertip output from the depth camera. In addition, various other methods can be used to acquire the three-dimensional position information of the fingertip.

(3) Generation and display of auxiliary information When the three-dimensional position coordinate information of the fingertip is acquired by the fingertip position information acquisition processing unit 11, the control unit 1 of the display control device CU generates auxiliary information under the control of the auxiliary information generation processing unit 12 as follows.

In other words, first, in step S14, the auxiliary information generation processing unit 12 calculates the distance in the depth direction between the aerial image RI and the user's fingertip based on the three-dimensional position coordinate information of the fingertip and the three-dimensional position coordinate information in the display space of the aerial image RI output from the aerial image display processing unit 13.

Then, in step S15, the auxiliary information generation processing unit 12 compares the calculated distance in the depth direction with a preset threshold value to determine whether the calculated distance is shorter than the threshold value. If the result of this determination is that the distance is equal to or greater than the threshold value, in step S16, auxiliary information including the auxiliary pattern and its display position coordinates is generated, and the generated auxiliary information is output to the aerial image display processing unit 13.

In step S17, the aerial image display processing unit 13 superimposes the auxiliary backturn output from the auxiliary information generation processing unit 12 onto the image of the display content based on its display position coordinates, and outputs it from the input/output I/F unit 4 to the display DP for display.

(3-1) First Display Example of Auxiliary Information FIG. 6 is a diagram showing a first display example of auxiliary information.
The auxiliary information generation processing unit 12 reads out an auxiliary pattern AI1 represented by an arrow as shown in Fig. 6 from the auxiliary pattern storage unit 31. The auxiliary pattern AI1 has an arrow shape in order to point to the aerial image that is the pointing target.

The auxiliary information generation processing unit 12 then sets display position coordinates to display the read arrow-shaped auxiliary pattern AI1 at the same depth position as the target TG that is the object of operation of the aerial image, on a plane perpendicular to the depth direction, at a position a preset distance Lb1 away from the center of the target TG. Then, auxiliary information including the auxiliary pattern and the display position coordinates is generated and passed to the aerial image display processing unit 13. Note that La1 indicates the distance in the depth direction between the center of the target TG of the aerial image RI and the user's fingertip.

The aerial image display processing unit 13 superimposes the auxiliary pattern AI1 contained in the auxiliary information on the video information of the display content so as to display the pattern at the position specified by the display position coordinates. The generated display video with the auxiliary pattern is then output from the input/output I/F unit 4 to the display DP.

As a result, in the display space, as shown in FIG. 6, an arrow-shaped auxiliary pattern AI1 is displayed at a position spaced a distance Lb1 from the center of the target TG along with the aerial image target TG.

The display position of the auxiliary pattern AI1 may be variably set so that the closer the distance La1 between the target TG and the fingertip, the closer the tip of the arrow is to the target TG. In this way, the user can smoothly bring his/her finger closer to the target TG depending on the display position of the auxiliary pattern AI1 relative to the target TG.

When setting the display position of the auxiliary pattern AI1, the aerial image display processing unit 13 determines the user's assumed viewpoint and takes into consideration the positional relationship between the determined assumed viewpoint and the position of the hand HD. For example, if the user's assumed viewpoint is above the hand HD, the display position of the auxiliary pattern AI1 is set above the display position of the target TG to prevent the hand HD from overlapping with the auxiliary pattern AI1.

By setting the display position of the auxiliary pattern AI1 in this way, even if the target TG in the aerial image completely overlaps with the fingertip, the user can refer to the auxiliary pattern AI1 and clearly grasp the depth position of the target TG without being affected by occlusion by the finger.

The user's expected viewpoint can be determined, for example, by capturing an image of the user from the side using a camera and recognizing the relative positions of the head and hands from the captured image.

(3-2) Second Display Example of Auxiliary Information FIG. 7 is a diagram showing a second display example of auxiliary information.
7 from the auxiliary pattern storage unit 31. Then, the auxiliary information generation processing unit 12 sets display position coordinates so as to display the read-out ring-shaped auxiliary pattern AI2 at a position concentric with the target TG of the aerial image and at a position where the radius Lb2 is proportional to the distance La2 between the center of the target TG and the fingertip. Then, auxiliary information including the auxiliary pattern AI2 and the display position coordinates is generated and output to the aerial image display processing unit 13.

The aerial image display processing unit 13 superimposes the auxiliary pattern AI2 of the auxiliary information on the video information of the display content based on the display position coordinates. Then, the display video on which the auxiliary pattern AI2 is superimposed is output from the input/output I/F unit 4 to the display DP.

As a result, as shown in FIG. 7, a ring-shaped auxiliary pattern AI2 having a radius Lb2 reflecting the distance La2 between the center of the target TG and the fingertip is displayed in the display space together with the target TG of the aerial image at a position surrounding the target TG. The radius Lb2 of this auxiliary pattern AI2 is set to become smaller, for example, as the user moves the fingertip closer to the target TG.

By displaying the auxiliary pattern AI2 as described above, the user can visually recognize the position of the target TG, which is the object of operation, on the display plane through the auxiliary pattern AI2, and can also accurately recognize the depth position of the fingertip relative to the target TG through the radius Lb2 of the auxiliary pattern AI2, i.e., the size of the ring, which allows the user to accurately and smoothly correct any misalignment of the user's finger.

(4) Pointing-Related Processing Meanwhile, suppose that the result of the determination in step S15 above is that the distance La2 between the aerial image RI and the user's fingertip is equal to or greater than a threshold value. In this case, the control unit 1 of the display control device CU determines that the user has performed a pointing operation on the target TG, and proceeds to step S19. Then, the control unit 1 of the display control device CU performs a process associated with the target TG of the aerial image RI that has been pointing-operated, such as turning a switch on and off or running an application.

The control unit 1 of the display control device CU monitors input of a display end request in step S18. If a display end request is not input, the control unit 1 returns to step S11 and executes the display control of the aerial image again in steps S11 to S18. Thereafter, the control of the aerial image display with auxiliary information in steps S11 to S18 is repeatedly executed in the same manner until a display end request is input.

(Action and Effects)
As described above, in one embodiment, while an aerial image is displayed in the display space, the three-dimensional position coordinates of the user's fingertip are acquired from the three-dimensional position sensor SU, and the distance in the depth direction between the acquired three-dimensional position coordinates of the fingertip and the target TG of the aerial image is calculated. Then, when the calculated distance is greater than or equal to a threshold value, auxiliary information is generated based on the auxiliary pattern and the distance, in which the distance is reflected in the display position of the auxiliary pattern, and the generated auxiliary information is displayed in the display space while being superimposed on the aerial image.

Therefore, according to one embodiment, even if the user has difficulty perceiving the positional relationship in the depth direction between his/her fingertip and the aerial image, the auxiliary information enables the user to perform an accurate and smooth pointing operation on the target in the aerial image.

In particular, by setting the display position of the auxiliary pattern taking into consideration the relationship between the user's hand position and the assumed viewpoint, as in the display example shown in Figure 6, the user can refer to the auxiliary pattern to grasp the depth position of the target TG without being affected by finger occlusion, and the impact of reduced depth perception due to inconsistencies in occlusion can be reduced.

In addition, as in the display example shown in Figure 7, by reflecting the distance between the user's fingertip and the target TG of the aerial image of the object to be operated in the display position of the auxiliary pattern, the user can correct the positional deviation of his or her fingertip in the depth direction by referring to the displayed auxiliary pattern, thereby reducing the effects of reduced depth perception due to a lack of accommodation.

[Other embodiments]
(1) In the embodiment, the display position of the auxiliary pattern with respect to the target TG of the aerial image or the radius of the auxiliary pattern is variably set according to the distance between the aerial image and the user's fingertip. However, the present invention is not limited to this. For example, a numerical value representing the distance between the aerial image and the user's fingertip or a bar representing the distance may be displayed instead of the auxiliary pattern or in addition to the auxiliary pattern.

(2) In one embodiment, an arrow pattern or a ring-shaped pattern is used as an auxiliary pattern. However, a pointing pattern or a rectangular frame pattern may also be used. The shape, color, size, etc. of the auxiliary pattern can also be set as appropriate.

(3) In one embodiment, a personal computer having the functions of the display control device CU is placed near the aerial image optical system DU. However, this is not limiting, and the functions of the display control device CU may be provided in a server computer placed on the web or in the cloud, for example, and this server computer may be configured to be able to communicate with the display DP and the three-dimensional position sensor SU via a network. In addition, the functions of the display control device CU may be distributed among multiple server computers or personal computers.

(4) The aerial image optical system may be configured, for example, using a retro-transparent material (MMAP), and the type and configuration of the aerial image that is the subject of the pointing operation, the type and configuration of the three-dimensional position sensor, and the processing procedures and processing contents of each processing unit of the display control device may be modified in various ways without departing from the spirit and scope of this invention.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all of the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

DU: Aerial image optical system DP: Display SU: Three-dimensional position sensor CU: Display control device 1: Control unit 2: Program storage unit 3: Data storage unit 4: Input/output I/F unit 5: Bus 11: Fingertip position information acquisition processing unit 12: Auxiliary information generation processing unit 13: Aerial image display processing unit 14: Display content acquisition processing unit 31: Auxiliary pattern storage unit 32: Display content storage unit

Claims (8)

a display control device that displays a display image on a display device;
an aerial image optical system that displays an optical image based on the display video displayed on the display device as an aerial image in a display space;
a position sensor that detects position information representing a position of a fingertip in the display space when a user performs a pointing operation on the aerial image,
The display control device includes:
a first processing unit that generates auxiliary information representing a positional relationship between the user's fingertip and the aerial image based on the position information detected by the position sensor and display position information of the aerial image in the display space;
a second processing unit that generates a display image with auxiliary information by superimposing the auxiliary information on the display image, and displays the generated display image with auxiliary information on the display device. A display control device used in an aerial image display system including an aerial image optical system that displays an optical image based on a display video displayed on a display device as an aerial image in a display space, and a position sensor that detects position information representing a position of a fingertip in the display space when a user performs a pointing operation on the aerial image,
a first processing unit that generates auxiliary information representing a first positional relationship between the user's fingertip and the aerial image based on the position information detected by the position sensor and display position information of the aerial image in the display space;
a second processing unit that generates a display image with auxiliary information by superimposing the auxiliary information on the display image, and displays the generated display image with auxiliary information on the display device. The display control device according to claim 2, wherein the first processing unit calculates the distance between the user's fingertip and the aerial image, and generates the auxiliary information when the calculated distance is greater than a preset threshold value. The display control device according to claim 2, wherein the first processing unit generates, as the auxiliary information, information including a first auxiliary pattern indicating the aerial image in the display space and display position coordinates for displaying the first auxiliary pattern within a certain distance range from the display position of the aerial image. The display control device according to claim 4, wherein the first processing unit determines a second positional relationship between the position of the user's fingertip relative to the aerial image and an assumed viewpoint, and sets the display position coordinates so as to display the first auxiliary pattern at a position that does not overlap with the user's fingertip based on the determined second positional relationship. The display control device according to claim 2, wherein the first processing unit calculates the distance between the user's fingertip and the aerial image, generates a second auxiliary pattern to represent the calculated distance, and sets the display position information so that the second auxiliary pattern is displayed at a position spaced apart from the display position of the aerial image by an interval corresponding to the distance. A display control method executed by a control device used in an aerial image display system including an aerial image optical system that displays an optical image based on a display video displayed on a display device as an aerial image in a display space, and a position sensor that detects position information representing a position of a fingertip in the display space when a user performs a pointing operation on the aerial image, comprising:
generating auxiliary information representing a positional relationship between the user's fingertip and the aerial image based on the position information detected by the position sensor and display position information of the aerial image in the display space;
generating a display image with auxiliary information by superimposing the auxiliary information on the display image, and displaying the generated display image with auxiliary information on the display device. A program that causes a processor provided in a display control device to execute the processing of at least one of the first processing unit and the second processing unit provided in the display control device according to any one of claims 2 to 6.

Images