WO2006118057A1 - Image display device - Google Patents

Abstract

An image display device enables a user to see a composite realistic-looking image without an uncomfortable feeling caused by that a virtual image is superposed on a real image. The image display device includes: a line-of-sight direction detection unit for detecting the line-of-sight direction of eyes of an observer; an imaging unit for capturing a real image arranged in the line-of-sight direction of user’s eye, a virtual image storage unit for storing a virtual image, a control unit for forming a combined image by superposing the virtual image stored in the virtual image storage unit on the real image captured by the imaging unit, and a display unit for displaying the combined image to the observer.

Description

 Specification

 Image display device

 Technical field

 The present invention relates to an image display device, and more particularly to an eyeball projection type image display device that displays a virtual image (including a video image) superimposed on a real image.

 Background art

 [0002] In recent years, a display device that fuses (superimposes) real and virtual images without a sense of incongruity in the technical field of mixed reality, which is a technology that fuses the real and virtual worlds. The importance of image display devices) is increasing.

 However, a conventional display device such as an optical see-through display has several problems. For example, a conventional display device has a narrow viewing angle at which a real image can be obtained. In addition, the brightness of the real image changes depending on the line of sight of the observer (user) (the object to be watched), whereas the brightness of the virtual image is constant, so the observer feels uncomfortable. In addition, since the virtual image related to the actual observed object (real image) 's distance to the point of gaze changes according to the observer's line-of-sight direction always looks the same, the perspective between the real image and the virtual image I can't get it, and I can't get mixed reality with more realism (verisimilar)! /.

 [0004] For example, according to a conventional display device proposed in Japanese Patent Laid-Open No. 8-313843 (Patent Document 1), a background image (wide-field image) displayed at a low resolution and a line of sight are followed. High-reality images can be realized by combining and displaying high-resolution narrow-field images that move. However, the above display device always displays a wide-field image that is related to the distance from the wide-field image and the narrow-field image to the observer (its eyeball). The distance between the object included in the object and the object included in the narrow field image cannot be accurately recognized. That is, according to the display device of Patent Document 1, it is not possible to express the perspective of an image depending on the difference in the distance (depth of field) in the viewing direction.

[0005] Also, Japanese Patent Application Laid-Open No. 11-313843 (Patent Document 2) describes a first video with a wide field of view and a high-resolution image. An image display device that synthesizes and displays a fine second image is disclosed. However, in the above video display device, it is suggested to display a virtual image with a depth of field of view that is always displayed clearly without depending on the lens effect of the eyeball. Further, in Patent Document 2, the depth of field of the virtual image is arbitrarily set by the observer, and the distance to the target object of the real image obtained by actual observation is compared with the depth of field. There is no mention of imagining and displaying virtual images without being arranged. Disclosure of the invention

 Problems to be solved by the invention

An object of the present invention is to provide an image display device that forms a composite image by appropriately fusing (superimposing) a virtual image with a real image and displaying the composite image without giving a sense of incongruity to an observer. And

 Means for solving the problem

[0007] An image display device according to one aspect of the present invention includes a gaze direction detection unit that detects a gaze direction of an observer's eyeball, and is disposed on the optical axis of the eyeball of the observer and is in the gaze direction of the eyeball. An image capturing unit that captures a real image, a virtual image storage unit that stores a virtual image, and a composite image in which a virtual image stored in the virtual image storage unit is superimposed on a real image captured by the image capturing unit. And a display unit for displaying a composite image to an observer.

 The invention's effect

 [0008] According to the image display device of one aspect of the present invention, a realistic composite image without a sense of incongruity in which a virtual image is superimposed on a real image is displayed, thereby realizing a more real mixed reality. can do.

 Brief Description of Drawings

FIG. 1 is a schematic diagram showing an image display apparatus according to Embodiment 1 of the present invention.

 2 is a block diagram showing each component unit of the image display device in FIG. 1. FIG.

 FIG. 3 is a schematic view showing a modification of the display unit of FIG.

FIG. 4 is a schematic view showing components of the display unit of FIG. FIG. 5 is a schematic diagram showing a process of calibrating the vertical center axis of the line-of-sight direction detection unit with respect to the line-of-sight direction in the image display device according to Embodiment 1.

 FIG. 6 is a schematic diagram showing a process of calibrating the vertical center axis of the line-of-sight direction detection unit with respect to the line-of-sight direction in the image display device according to Embodiment 1.

 FIG. 7 is a schematic view showing another modification of the display unit of FIG.

 FIG. 8 is a schematic view showing still another modification of the display unit of FIG.

 FIG. 9 is a schematic view showing still another modification of the display unit of FIG.

 FIG. 10 is a schematic view showing a modification of the drive unit in FIG. 1.

 FIG. 11 is a schematic diagram showing a process of removing the drive unit of FIG. 1 from the line-of-sight direction when a failure occurs in the image display apparatus according to Embodiment 1.

 FIG. 12 is a schematic diagram showing an image display apparatus according to Embodiment 2 of the present invention.

 FIG. 13 is a schematic diagram showing an image display apparatus according to Embodiment 3 of the present invention.

 FIG. 14 is a schematic diagram showing an arrangement position of an eyeball, a real image, and a virtual image when an observer observes a real image and a virtual image at the same time using the image display device of the third embodiment.

 FIG. 15 is a schematic diagram showing an optical system that realizes Maxwellian vision using a point light source (directional light source) employed in an image display apparatus according to Embodiment 4 of the present invention.

 FIG. 16 is a schematic diagram showing an optical system using a normal surface light source (diffusive light source).

Explanation of symbols

10: Display unit

 11: Housing, 12: LCD panel (display unit), 14: First mirror (optical component), 16: Eyeball camera (line-of-sight detection unit), 18: Camera for line-of-sight shooting (imaging unit), 20 : Second mirror, 22: Light source, 23: Infrared light source, 24: Eyepiece, 25: Correction lens, 32: Optical shirt,

 40: Drive unit

42: first pivot (rotary drive), 44: second pivot, 46: first arm, 48: second arm, 50: hollow guide housing, 52: electromagnet, 54: Permanent magnet, 56: Outer wall, 57: Inner wall, 58: End wall, 60: Information processing unit

 62: Control unit, 64: Non-volatile memory (virtual image storage unit), 66: Input unit,

 LS: Gaze direction, EB: Eyeball, CB: Calibration signal (light flux), GP: Gaze point, RI: Actual image, VI: Virtual image.

 BEST MODE FOR CARRYING OUT THE INVENTION

 Hereinafter, an embodiment of an image display device according to the present invention will be described with reference to the accompanying drawings.

 In the description of the embodiment, a force that appropriately uses terms indicating directions (for example, “right” and “left”) to facilitate understanding. This is for explanation, and these terms are It is not intended to limit the invention.

 [0012] Embodiment 1.

 A first embodiment of the image display apparatus according to the present invention will be described below with reference to FIGS. FIG. 1 is a schematic diagram showing an image display device 1 according to Embodiment 1 of the present invention, and FIG. 2 is a block diagram showing each component unit of the image display device 1 of FIG. Note that this image display device is normally used by being attached to the observer's eyes (left eye and right eye). In the following embodiments, the image display device attached to the left eye unless otherwise specified. It is to be understood that the image display apparatus mounted on the right eye is configured symmetrically with respect to the symmetry plane SS indicated by the one-dot chain line in FIG.

 As shown schematically in FIGS. 1 and 2, an image display device 1 according to the present embodiment includes a display unit 10, a drive unit 40 that moves the display unit 10, and the display unit 10 and the drive unit 40. And an information processing unit 60 connected to be wired or wirelessly controllable.

In FIG. 1, the display unit 10 includes a display unit 12 such as a liquid crystal display (LCD) panel disposed inside a housing 11 having a substantially hemispherical outer shape, and an optical component 14 such as a semitransparent mirror. And an eye photography camera (line-of-sight detection unit) 16 for detecting the line-of-sight direction LS of the observer (user) indicated by a broken-line arrow in FIG. Further, the display unit 10 includes a camera (imaging unit) 18 for capturing a visual line direction for capturing an actual image observed in the visual line direction LS. According to the configuration shown in FIG. 1, the imaging unit 18 is located outside the nosing / housing 11 and is disposed in the line-of-sight direction LS. In addition, the display unit 12, the optical component 14, the line-of-sight direction detection unit 16, and the imaging unit 18 constituting the display unit 10 are fixed with respect to each other in the housing 11, and as the housing 11 moves, It is possible to move together while maintaining the relative positional relationship.

 Note that these cameras (the line-of-sight direction detection unit 16 and the imaging unit 18) process the obtained image information digitally by the information processing unit 60, and are preferably a charge-coupled device (CCD) or CMOS imaging unit. It is configured using a digital image sensor such as a device.

 Similarly, referring to FIG. 1, the drive unit 40 includes a first pivot part (rotary drive part) 42 connected to the housing 11 of the display unit 10 at one end and a second part at the other end. And a second arm 48 slidably attached to the second pivot 44. The first arm 46 (including the first and second pivot portions 42 and 44) and the second arm 48 move the display unit 10 to an arbitrary position in response to a command from the information processing unit 60. Can be made.

 [0016] The information processing unit 60 is a general information control terminal such as a personal 'computer or a portable' computer, and includes a control unit 62 such as a central processing unit (CPU) and a hard disk for storing virtual images or It has a virtual image storage unit 64 composed of nonvolatile memory such as a flash memory and an input unit 66 composed of a keyboard or touch panel.

[0017] When the display unit 10 is attached to an observer (user), as described above, the line-of-sight direction detection unit 16 passes through the semi-transparent mirror 14 and the pupil position of the observer's eyeball EB, that is, the observer's line of sight. The direction LS is detected, and the line-of-sight direction data is transmitted to the control unit 62 of the information processing unit 60. Then, the control unit 62 drives the drive unit 40 to move the display unit 10 so that the imaging unit 18 is arranged in the detected line-of-sight direction LS. Since a series of operations of the line-of-sight direction detection unit 16, the information processing unit 60, and the drive unit 40 is continuously performed while the display unit 10 is mounted on the observer, the imaging unit 18 always observes. A real image of the actual object observed in the person's gaze direction LS is captured. Real image data captured by the imaging unit 18 is sequentially transmitted to the control unit 62. [0018] Similarly, the virtual image data stored in the virtual image storage unit 64 is transmitted to the control unit 62, where it is digitally processed and superimposed on the real image captured by the imaging unit 18, Data of the synthesized image is generated. Further, the composite image data is transmitted to the display unit 10, displayed on the display unit 12, and the light beam passing therethrough is reflected by the mirror 14 and projected onto the observer's eyeball EB. Thus, according to Embodiment 1, it is possible to appropriately combine a virtual image with a real image and provide a realistic composite image to an observer. At this time, the control unit 62 adjusts the brightness of the real image and the virtual image to generate composite image data, so that a more natural composite image can be displayed to the observer.

 The line-of-sight direction detection unit 16 constantly monitors the line-of-sight direction of the observer and transmits line-of-sight direction data to the control unit 62. Therefore, when the line of sight of the observer moves, the control unit 62 A command is issued to 40, and the display unit 10 is moved so that the imaging unit 18 captures an image in the line-of-sight direction LS. At this time, preferably, the drive unit 40 moves the display unit 10 on a spherical surface centered on the eyeball EB of the observer so that the distance to the display unit 10 is constant also in the eyeball EB force of the observer. . As a result, according to the image display device 1 according to the first embodiment, the real image always in the line-of-sight direction LS is captured following the observer's line-of-sight direction LS, and the composite image is superimposed on the virtual image. Since the image can be displayed to the observer, a more natural mixed reality can be given to the observer.

 [0020] Hereinafter, detailed configurations (including modifications) and operations of each component of the first embodiment will be described in detail, but other configurations that can be easily understood by those skilled in the art are similarly described. Is understood to be included in the present invention.

The imaging unit 18 of the display unit 10 shown in FIG. 1 is located outside the housing 11 and is disposed on the line-of-sight direction LS, but is not limited to this. That is, in the image display device 1 shown in FIG. 3, the imaging unit 18 is disposed inside the housing 11. In addition to the first mirror 14 that reflects the image displayed on the display unit 12 toward the observer's eyeball EB, a real image in the line-of-sight direction LS is projected onto the imaging unit 18 in the nosing 11. A second mirror 20 is provided. Thus, the real image in the line-of-sight direction is picked up by the image pickup unit 18 and superimposed on the virtual image by the control unit 62 as in the first embodiment. The composite image is displayed on the display unit 12, reflected by the first mirror 14, and displayed to the observer. Is done.

 [0022] The first mirror 14 may be a force deflection separation mirror described as being translucent. The second mirror 20 may be a dichroic mirror that reflects visible light and transmits infrared light.

 As shown more specifically in FIG. 4, the display unit 10 includes a light source 22, a transmissive LCD panel (display unit) 12 disposed adjacent to the light source 22, and a light flux from the LCD panel 12. It has a deflecting beam splitter (optical component) 14 that reflects toward the eyeball EB, and an eyepiece 24 that focuses the light beam onto the eyeball EB. The line-of-sight direction detection unit 16 and the imaging unit 18 are assembled so that their vertical center axes are arranged in a straight line.

 In the display unit 10 configured as described above, white light from the light source 22 is transmitted through the transmissive LCD panel 12, and the light beam including the composite image displayed on the transmissive LCD panel 12 is reflected by the deflecting beam splitter 16. And projected onto the observer's eyeball EB. As a result, the observer can see the composite image displayed on the transmissive LCD panel 12.

 [0024] As described above, according to the image display device 1 according to the first embodiment, it is possible to follow the observer's line of sight, capture a real image always in the line of sight, and display it to the observer. it can. In order to realize this, as shown in FIG. 1, it is necessary to accurately align the vertical center axes of the line-of-sight direction detection unit 16 and the imaging unit 18 with the line-of-sight direction LS. The gaze direction detection unit 16 and the imaging unit 18 are easy to assemble (fix) so that their vertical central axes are arranged in a straight line. It is difficult to accurately align the vertical center axis of the line-of-sight direction detection unit 16 (imaging unit 18) and the line-of-sight direction LS. It is necessary to perform a process (calibration) that reliably calibrates the central axis to the line-of-sight direction LS.

[0025] In order to accurately align the vertical center axis of the line-of-sight direction detection unit 16 and the line-of-sight direction LS, as shown in Fig. 5, the LCD panel 12 An image of a predetermined shape such as a circle is displayed at the center of the image sensor 13 (ie, the center of the imaging unit 18), and a calibration signal (light beam) CB is projected to the translucent mirror 14 by directing it (arrow 26a). Then, it is reflected by the semitransparent mirror 14 and irradiated to the eyeball EB (arrow 26b). At this time, the calibration light beam CB is reflected by the eyeball EB in the line-of-sight direction LS, passes through the translucent mirror 14, It is detected by the line-of-sight direction detector 16 (arrow 26c). That is, if the center of the line-of-sight detection unit 16 (calibration signal CB of a predetermined shape) and the line-of-sight direction (intersection 28 of the dashed line in FIGS. 6 (a) and (b)) LS match, FIG. As shown in a), a calibration-shaped luminous flux (image) with a predetermined shape is detected at the intersection 28 of the dotted chain line, but if the central axis of the line-of-sight direction detector 16 and the line-of-sight direction LS are not accurately aligned, As shown in Fig. 6 (b), the image of the predetermined shape deviates from the intersection 28 of the alternate long and short dash line.

 [0026] As shown in FIG. 6 (b), when the image (calibration light beam) CB of a predetermined shape deviates from the intersection 28 of the dashed-dotted line, the line-of-sight direction detection unit 16 detects that in FIG. Based on the image, the control unit 62 calculates a deviation amount of the center axis (that is, the calibration light beam CB) of the imaging unit 18 with respect to the intersection 28 (that is, the line-of-sight direction LS) of the one-dot chain line. Then, the control unit 62 moves the display unit 10 using the drive unit 40 in accordance with the amount of deviation of the central axes of the imaging unit 18 and the gaze direction detection unit 16 with respect to the gaze direction LS, as shown in FIG. Calibrate to such an aligned condition. Such an image having a predetermined shape (calibration light beam CB) is preferably projected onto the eyeball EB at an interval shorter than the time recognizable by the observer when the observer observes the predetermined gazing point. This makes it possible to calibrate the central axis of the gaze direction detection unit 16 (and the imaging unit 18) to the gaze direction LS without bothering the observer as much as possible. More preferably, such a calibration process is periodically performed so that the central axis of the line-of-sight direction detection unit 16 and the line-of-sight direction LS are always aligned.

 In the state where the central axis of the line-of-sight direction detection unit 16 and the line-of-sight direction LS are aligned, the line-of-sight direction detection unit 16 constantly monitors the line-of-sight direction LS. Specifically, as shown in FIG. 7, the line-of-sight direction detection unit 16 has an infrared light source 23 such as an infrared LED lamp, and the infrared light from the infrared light source 23 passes through the translucent mirror 16. Then, the pupil position of the eyeball EB, that is, the line-of-sight direction LS is detected by detecting the reflected infrared ray irradiated to the eyeball EB of the observer. As described above, according to Embodiment 1, the line-of-sight direction LS is detected using infrared light that is not recognized by the observer, so the observer's line-of-sight direction LS is constantly monitored without giving the observer a sense of incongruity. be able to.

[0028] The display unit 10 described above with reference to FIG. 7 includes a gaze direction photographing camera (imaging unit) 18 for capturing an actual image observed in the gaze direction LS, and the gaze direction of the observer. Inspection As shown in FIG. 8, the present invention is not limited to this. However, the present invention is not limited to this, and the imaging unit 18 and the line-of-sight detection unit 1 6 , And a single CCD or CMOS imaging device 30 may be used to image both the real image and the pupil of the eyeball EB. The image pickup device 30 is generally formed using a silicon substrate, and is configured to pick up a real image with the front side facing the line-of-sight direction LS and detect infrared light irradiated from the back side. can do. In this case, as shown in FIG. 8, an optical shirter 32 such as a liquid crystal shirter is arranged in front of the imaging unit 18, and the optical shirter 32 is opened to pick up a real image, and the line of sight is detected. Closes the optical shirt 32. In order to detect the line-of-sight direction LS, it is preferable to shorten the time during which the optical shirt 32 is closed as much as possible and detect the line-of-sight direction LS periodically. As a result, the image display apparatus 1 can always display the real image in the line-of-sight direction LS to the observer following the line of sight of the observer.

 In this way, when both the real image and the eyeball EB are imaged using the single imaging device 30, the gaze direction detection unit 16 and its peripheral circuit (image forming circuit) can be shared (omitted). Therefore, the number of parts can be reduced and the display unit 10 can be reduced in size and weight.

[0029] In the first embodiment, a force that employs a transmissive LCD panel as the display unit 12 is not limited to this. That is, the display unit 10 includes a light source 22 and a display unit 12 such as a reflective LCD panel, as shown in FIG. That is, the white light having the light source 22 passes through the deflecting beam splitter 14, and the light beam formed as an image by the reflective LCD panel 12 is reflected by the deflecting beam splitter 14 and irradiated to the observer's eyeball EB. Thus, the observer can observe the composite image displayed on the reflective LCD panel 12. Since the display unit 10 including the reflective LCD panel 12 can generally achieve higher definition image quality than the display unit including the transmissive LCD panel 12, the observer can use the reflective display unit 10 shown in FIG. It can be used to observe a more precise image.

Furthermore, in Embodiment 1, the drive unit 40 may have the force and other structures described as having the first and second arms 42 and 44. For example, as shown in FIG. 10, the drive unit 40 is roughly composed of a substantially hemispherical outer peripheral wall 56 and a transparent member. And a hollow guide housing 50 formed by the inner peripheral wall 57 and the end wall 58, and the display unit 10 is slidably disposed in the hollow guide housing 50. In the vicinity of the end wall 58 of the hollow guide housing 50, a plurality of electromagnets 52 are disposed, while a plurality of permanent magnets 54 are disposed in the display unit 10. The control unit 62 adjusts the magnetic force (the amount of current flowing through the electromagnet 52) of each electromagnet 52 in the hollow guide nosing 50 so that the imaging unit 18 is arranged on the detected line-of-sight direction LS. The display unit 10 can be moved along the hollow guide nosing 50 (arrow 56).

 [0031] Furthermore, in the image display device 1 according to the first embodiment, the control unit 62 has failed in the display unit 10 or the information processing unit 60 and cannot display an appropriate composite image to the observer. When this is detected, it is preferable to drive the first arm 46 to remove the display unit 10 from the observer's line-of-sight direction LS as shown in FIGS. 11 (a) and 11 (b). This will restore the observer's naked eye field of view and ensure the safety of the observer. The failure of the display unit 10 and the information processing unit 60 is not limited to this, but for example, interruption of image data transmission from the line-of-sight direction detection unit 16 and the imaging unit 18 to the control unit 62, malfunction of the display unit 12 The power failure or malfunction of the control unit 62 is included. Further, in order to exclude the display unit 10 from the observer's line-of-sight direction LS, a force that can be realized by using any means understood by those skilled in the art. For example, the display unit 10 is energized to exclude the display unit 10. A spring member (not shown) is released in response to the failure, and the state force of FIG. 11 (a) also moves the first arm 46 to the state of FIG. 11 (b).

 [0032] Embodiment 2.

 Embodiment 2 of the image display apparatus according to the present invention will be described below with reference to FIG. In the image display device 2 of the second embodiment, the real image is directly observed by the observer, and the display unit 10 is the image of the first embodiment except that only the virtual image is projected onto the observer's eyeball EB. Since it has the same configuration as that of the display device 1, detailed description of the overlapping components is omitted. The same components as those in Embodiment 1 will be described using the same reference numerals.

[0033] The display unit 10 according to the second embodiment generally includes a light source 22 disposed inside the housing 11, a display unit 12 such as a liquid crystal display (LCD) panel, and reflects visible light by reflecting infrared light. Transmitting dichroic mirror 15; Translucent mirror 14 that reflects infrared light and transmits visible light; Eyepiece 24 and correction lens 2 5 for converging LS-direction luminous flux into eyeball EB, observer A line-of-sight direction detection unit 16 for detecting the line-of-sight direction LS, and an imaging unit 18 for measuring the brightness of an actual image observed in the line-of-sight direction LS.

 [0034] As described above, the observer can directly see the object in the line-of-sight direction LS through the eyepiece lens 25, the translucent mirror 14, and the correction lens 24. Further, the luminous flux with the line-of-sight direction LS force is reflected by the semi-transparent mirror 14, and the amount of light (the actual brightness of the image observed in the line-of-sight direction LS) is detected by the imaging unit 18.

 The line-of-sight direction detection unit 16 has an infrared light source 23. Infrared light from the infrared light source 23 is reflected by the dichroic mirror 15 and the semitransparent mirror 14 and projected onto the eyeball EB. It is detected by the detector 16. Thus, as in the first embodiment, the line-of-sight direction detection unit 16 can detect the line-of-sight direction LS of the eyeball EB of the observer.

 According to the second embodiment, the virtual image stored in the non-volatile memory 64 of the processing arithmetic unit 60 is processed by the control unit 62 and then transmitted to the LCD panel 12 to form an image. Then, the white light from the light source 22 passes through the LCD panel 12, and the light beam including the virtual image passes through the dichroic mirror 15, is reflected by the semitransparent mirror 14, and is projected onto the observer's eyeball EB. Thus, the observer can observe the virtual image while directly viewing the real image. At this time, the light amount of the real image (the actual brightness of the image observed in the line-of-sight direction LS) is detected by the imaging unit 18, and the light amount data is transmitted to the control unit 62. Can be adjusted according to the brightness of the real image. Thereby, according to the image display device of the second embodiment, the virtual image can be superimposed on the real image more naturally and displayed to the observer.

 Embodiment 3.

A third embodiment of the image display apparatus according to the present invention will be described below with reference to FIGS. The image display device 3 of the third embodiment has the same configuration as the image display devices 1 and 2 described above except that it has a distance sensor that measures the distance from the eyeball EB to the gazing point. Detailed description of the constituent elements to be performed will be omitted. Also, Components similar to those in the first embodiment will be described using the same reference numerals.

[0037] A gaze point is generally defined as a point where an object to be observed by an observer exists, but when observing an object with both eyes, the gaze point GP is This is also the intersection of the eyes LS and LS. In Fig. 13, the left and right eyeballs EB, the line segment between the centers of EB C

L R L R L

 — The distance L of C is unique to the observer. The center of each eyeball EB, EB

R 0 L R

 C, C force Gaze point The angle between the straight line extending to GP and the center line segment C — C is

L R L R

 Each eyeball EB can be calculated from the line-of-sight direction LS, for example, 0 and Θ.

 1 2

 The That is, the distances L and L from the gazing point GP to the left and right EB and the centers C and C of EB are

 L R L R 1 2 can be expressed as a function of distance L and angles θ and Θ.

 0 1 2

 [0038] In general, when an observer observes a nearby object with the naked eye (focusing) and observes another object far away on the extended line of the line of sight, The object appears to be clear, while objects farther away (or closer) are perceived as blurred. This allows the observer to sense the perspective of two objects at different distances.

 [0039] Image display device 3 according to Embodiment 3 is configured to reproduce the natural perspective obtained when observed with the naked eye as described above. That is, in FIG. 14, the observer is gazing at the chair (real image) RI at the gazing point GP, and the distance L from the eyeball EB to the gazing point is calculated as described above. At this time, when forming a composite image by superimposing a virtual image VI (image of a pigeon in FIG. 14) VI on the chair's real image RI that is assumed to be distant from the gazing point GP, the control unit 62 Calculates the real image data obtained from the imaging unit 18 and the virtual image data stored in the virtual image storage unit 64 to display the real image RI clearly, while displaying the virtual image VI in a blurred manner. Thus, a composite image is formed.

 [0040] More specifically, in the third embodiment, attribute information X related to the distance (virtual distance) to the position where the virtual image VI should be seen (the position where it should be seen) is assigned to the virtual image VI. The virtual image VI is stored in the nonvolatile memory 64. Alternatively, the observer may input the attribute information of the virtual distance X using the input unit 66 of the information processing unit 60. That is, each virtual image can have an arbitrary virtual distance (attribute information) X.

Then, the information processing unit 60 of Embodiment 3 determines the distance to the real image RI (the real distance ) Measure L and compare the distance (virtual distance X) to the position where the virtual image VI should be visible (the position where it will be visible) .If the real distance L and the virtual distance X are different, the virtual image is Real image RI is clearly displayed and virtual image VI is unclearly displayed when the image is far from or near the image.

 [0041] For example, when the virtual distance X to the position where the virtual image (dove) VI would be visible is set to 10m, the distance L to the gaze point GP of the real image (chair) RI is measured at 5m. Let's assume that. At this time, when the observer observes the virtual image (pigeon) VI 10m ahead and the real image (chair) RI seen 5m ahead with the same sharpness, it feels strange because it is different from the normal sense of distance. Therefore, the information processing unit 60 according to Embodiment 3 compares the virtual distance X with the real distance, and blurs the virtual image VI that is not placed near the real image (gaze point GP) in the gaze direction LS. Real image Superimposes on RI to form a composite image. The synthesized image obtained in this way allows the observer to obtain a more realistic perspective.

 [0042] Further, the information processing unit 60, when the virtual distance X is within a predetermined distance d with respect to the actual distance L (that is, when X becomes L−d or X> L + d). ) Alternatively, only the virtual image VI may be blurred and superimposed on the real image RI and displayed on the LCD panel 12.

 [0043] In the above description, the left and right EBs, the distance L between the EB center line segments C C,

 L R L R 0

 Using the angles 0 and Θ between the eyeball EB and the center line segment C — C,

 L R 1 2

 The distances L and L to the centers C and C of EB and EB were obtained, but any distance sensor is displayed.

L R L R 1 2

 It may be installed in unit 10 and the distance from the point of sight GP to the eyeball EB may be measured directly. In this case, use one of the display units 10 for the left and right eyeballs EB and EB.

 L R

 The perspective of the virtual image VI with respect to the real image RI can be expressed. That is, the observer can enjoy a realistic image with only one eye using the display unit 10 having a simpler configuration.

Similarly, in the image display apparatus 2 according to Embodiment 2 in which only the virtual image VI is displayed on the display unit 10 and the real image RI is observed with the naked eye, the visual distance L is compared with the real distance L and the virtual distance X. It is not placed near the real image (gaze point GP) in the direction LS! ヽ It is possible to provide the observer with a realistic virtual image VI by blurring the virtual image VI and projecting it on the eyeball EB. it can. [0044] Embodiment 4.

 Embodiment 4 of the image display device according to the present invention will be described below with reference to FIGS. Whereas the light source of the display unit 10 of the image display device described so far is a surface light source, the image display device 4 of the fourth embodiment is generally except that the display unit 10 includes a point light source. Since it has the same configuration as that of the image display apparatus described so far, detailed description of the overlapping components will be omitted. The same components as those in the above embodiment are described using the same reference numerals.

 As described above, the light source 22 of the display unit 10 of Embodiment 4 shown in FIG. 15 is configured to be a point light source (a light source that emits a light beam having directivity), and the line-of-sight direction of the light source 22 The size in the direction orthogonal to LS (width in the line-of-sight direction) w is configured to have the same size as the pupil diameter a of the eyeball EB. More specifically, the light source 22 is set so that its size w is 4 times or less, preferably 2 times or less of the pupil diameter a, and when the general pupil diameter a is 3 mm, The size w of 22 is set to be 12 mm or less, preferably 6 mm or less.

 When the light source 22 of the display unit 10 is set as described above, the light flux from the light source 22 is uniformly diffused from a limited area (point light source) as shown in FIG. The light passes through 12 and is condensed by the condenser lens 34 so as to focus on the pupil P of the eyeball EB, and forms an image on the retina. At this time, an observer without depending on the lens effect of the lens CL of the eyeball EB can always see a clear image (Maxwell's view). In other words, even if the distance (depth of focus) from the observer's eyeball EB to the gazing point changes, the image displayed on the LCD panel 12 should be projected directly (with good reproducibility) onto the observer's eyeball EB. Can do.

 On the other hand, when the light source 22 is configured as a surface light source adjacent to the LCD panel 12 (a light source that emits a light beam having diffusibility) (see FIG. 16), the lens CL has a predetermined lens effect ( The observer can perceive a clear image only when it has a lens power.

[0048] Therefore, when the optical system that realizes Maxwellian vision described in the fourth embodiment is applied to the image display device 1 of the first embodiment, the combined image power SLCD in which the virtual image VI is superimposed on the real image RI. Displayed on the panel 12, the observer can clearly see the composite image displayed on the LCD panel 12 regardless of the real distance L and the virtual distance X. Similarly, when an optical system that realizes Maxwell's vision is used in the image display device 2 of Embodiment 2, it is actually related to the actual distance L (depth of focus) when viewing a real image with the naked eye. Regardless of whether the observed object is far or near, the virtual image displayed on the LCD panel 12 can always be clearly recognized without depending on the lens effect of the lens CL of the eyeball EB. it can.

 In other words, as in the third embodiment, the observer sets an arbitrary value using the input unit 66 using the virtual distance X as attribute information of the virtual image VI, and the control unit 62 uses the LCD panel 12. When the real distance L and the virtual distance X are different, the real image RI may be displayed clearly and the virtual image VI may be displayed unclearly.

Claims

The scope of the claims  [1] A gaze direction detection unit that detects the gaze direction of the eyeball of the observer,  An imaging unit that is arranged in the line-of-sight direction and captures a real image in the direction of the eyeball; a virtual image storage unit that stores a virtual image;  A control unit that superimposes the virtual image stored in the virtual image storage unit on the real image captured by the imaging unit to form a composite image;  An image display device comprising: a display unit that displays a composite image to an observer.  [2] The line-of-sight direction detection unit, the imaging unit, and the display unit are fixedly arranged with respect to each other so that when the observer's eyeball moves, the line-of-sight direction matches the optical axis of the imaging unit. 2. The image display device according to claim 1, further comprising a drive unit that moves the line-of-sight direction detection unit, the imaging unit, and the display unit. [3] The image display device according to claim 1, wherein the line-of-sight direction detection unit detects the line-of-sight direction using infrared rays. 4. The image display device according to claim 1, wherein the line-of-sight direction detection unit and the imaging unit are configured by a single imaging device. 5. The display unit according to claim 1, wherein the display unit includes a light source and an image forming device, and reflects light from the light source by the image forming device to display a composite image to an observer. The image display device described.  [6] The line-of-sight direction detection unit, the imaging unit, and the display unit are respectively arranged for the left and right eyeballs of the observer,  The control unit determines the distance between the left and right eyeballs and the gazing point from the gaze direction of the left and right eyeballs detected by the gaze direction detection unit for the left and right eyeballs and the distance between the left and right eyeballs. 2. The image display device according to claim 1, wherein measurement is performed. 7. The display unit according to claim 6, wherein the display unit cooperates with the control unit to blur and display a virtual image that is not arranged near the gazing point in the gaze direction! Image display device [8] It further has a distance measuring unit that measures the distance between the eyeball and the gazing point, The display unit is arranged in the vicinity of the gazing point in the line-of-sight direction in cooperation with the control unit. The image display device according to claim 1, wherein the virtual image is displayed in a blurred manner. [9] The display unit includes a point light source, an image forming device, and a condenser lens.  By transmitting the light from the point light source to the image forming device and condensing it on the observer's eyeball with the condenser lens, the observer can always obtain a clear image without depending on the lens effect of the eyeball. The image display device according to claim 1, wherein the image display device is perceived. [10] The line-of-sight direction detection unit, the imaging unit, and the display unit are respectively arranged for the left and right eyeballs of the observer,  The control unit measures a distance between the left and right eyeballs and a gazing point from the gaze direction of the left and right eyeballs detected by the gaze direction detection unit for the left and right eyeballs. 9. The image display device according to 9. [11] An input unit connected to the control unit for inputting attribute information regarding a virtual position where a virtual image can be clearly seen,  10. The image display device according to claim 9, wherein the display unit blurs and displays a virtual image in which a virtual position is not arranged in the vicinity of the gazing point in the gaze direction in cooperation with the control unit.  [12] An input unit connected to the control unit for inputting attribute information regarding a virtual position where a virtual image is clearly visible;  A distance measuring unit that measures the distance between the eyeball and the gazing point;  10. The image display device according to claim 9, wherein the display unit blurs and displays a virtual image in which a virtual position is not arranged in the vicinity of the gazing point in the gaze direction in cooperation with the control unit.  13. The image display device according to claim 8, wherein the size of the point light source is not more than four times the size of the pupil of the eyeball of the observer in a direction orthogonal to the line-of-sight direction.  14. The image display device according to claim 1, wherein when an obstacle occurs in the operation of the image display device, the imaging unit is excluded from the line-of-sight direction of the observer's eyeball.  [15] a virtual image storage unit for storing virtual images; A display unit for displaying a virtual image; And an optical component that is arranged on the optical axis of the eyeball of the observer and that superimposes the virtual image displayed on the display unit on the real image in the eye gaze direction and forms an image on the eyeball. An image display device.  16. The image display device according to claim 15, wherein the optical component includes a translucent mirror that transmits a real image and reflects a virtual image. 17. The image display device according to claim 15, further comprising: a gaze direction detection unit that detects a gaze direction of the eyeball of the observer. 18. The image display device according to claim 17, wherein the line-of-sight direction detection unit detects the line-of-sight direction of the observer's eyeball using infrared rays. [19] The display unit includes a light source and an image forming device, and reflects light from the light source by the image forming device to display a virtual image to an observer. The image display device described. [20] The display unit includes a point light source, an image forming device, and a condenser lens,  By transmitting the light from the point light source to the image forming device and condensing it on the observer's eyeball with the condenser lens, the observer can always obtain a clear image without depending on the lens effect of the eyeball. 16. The image display device according to claim 15, wherein the image display device is perceived. [21] An input unit connected to the control unit for inputting attribute information regarding a virtual position where a virtual image is clearly visible;  A distance measuring unit that measures the distance between the eyeball and the gazing point;  21. The image display device according to claim 20, wherein the display unit blurs and displays a virtual image in which the virtual position is not arranged in the vicinity of the gazing point in the gaze direction in cooperation with the control unit.