WO2025243410A1 - Semiconductor device, method, and head-mounted display - Google Patents

Abstract

This semiconductor device includes a memory, a display data generation circuit, and a display control circuit. In the memory, first positional relationship data is stored, the data indicating a positional relationship between a subject included in a transmission image transmitted through a transmissive display and the subject included in first image data captured by a first camera that captures an image of a direction in which the back face of the transmissive display faces. The display data generation circuit uses the first positional relationship data to extract and deform at least a partial region of the first image data, and to generate display data such that the contour of the subject included in the transmission image is continuous with the contour of the subject included in the first image data region on the transmissive display. The display control circuit displays the display data on the transmissive display.

Description

Semiconductor device, method, and head-mounted display

The present invention relates to a semiconductor device, a method, and a head-mounted display.

Conventionally, technologies such as head-mounted displays and head-up displays are known that superimpose images captured by a camera onto a see-through display.

With this technology, the user can simultaneously view both the transmitted image through the transmissive display and the captured image displayed on the transmissive display.

Patent No. 7246708 Japanese Patent Application Laid-Open No. 2008-96867 Patent No. 5855206

However, when a captured image is displayed on a transmissive display, visibility may be reduced at the boundary between the transmissive image and the captured image.

In one aspect, the present invention aims to improve the visibility of the boundary between a transmitted image and a captured image when the captured image is displayed on a transmissive display.

The semiconductor device of the embodiment includes a memory, a display data generation circuit, and a display control circuit. The memory stores first positional relationship data indicating the positional relationship between a subject included in a transmitted image transmitted through the transmissive display and a subject included in first image data captured by a first camera that captures the direction in which the back of the transmissive display faces. The display data generation circuit extracts and deforms at least a partial area of the first image data based on the first positional relationship data, and generates display data such that the outline of the subject included in the transmitted image and the outline of the subject included in the area of the first image data are continuous on the transmissive display. The display control circuit displays the display data on the transmissive display.

FIG. 1 is a diagram illustrating an example of the overall configuration of a head-mounted display according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an SoC according to the first embodiment. FIG. 3 is a diagram showing an example of the difference between a video see-through display image and a transmission image according to the first embodiment. FIG. 4 is a diagram showing an example of the installation position of the calibration camera according to the first embodiment. FIG. 5 is a diagram illustrating an example of a subject for the calibration process according to the first embodiment. FIG. 6 is a diagram showing an example of a subject 9b photographed from the side by the calibration camera according to the first embodiment. FIG. 7 is a flowchart showing an example of the flow of the calibration process according to the first embodiment. FIG. 8 is a diagram showing an example of the processing content of each step in the flowchart of FIG. FIG. 9 is a diagram showing an example of distortion contained in each image data in the calibration process according to the first embodiment. FIG. 10 is a diagram illustrating an example of distortion contained in each image data when the head-mounted display according to the first embodiment is used. FIG. 11 is a diagram showing an example of the breakdown of the modification process B in FIG. FIG. 12 is a flowchart showing an example of the flow of a process for generating display data when the head-mounted display according to the first embodiment is used. FIG. 13 is a diagram showing an example of correction of video see-through image data according to the first embodiment. FIG. 14 is a diagram illustrating the principle of visual assistance using the transmissive display and liquid crystal shutter according to the first embodiment. FIG. 15 is a diagram illustrating an example of a display mode of the transmissive display according to the first embodiment. FIG. 16 is a diagram showing an example of the flow of the bright place/dark place region extraction process according to the first embodiment. FIG. 17 is a diagram illustrating an example of the flow of the EV value acquisition process according to the first embodiment. FIG. 18 is a diagram showing an example of a threshold value determined based on a histogram according to the first embodiment. FIG. 19 is a diagram showing an example of a general EV/Lux conversion table. FIG. 20 is a diagram showing an example of the relationship between the EV value and the transmittance of the lens according to the first embodiment. FIG. 21 is a diagram showing an example of a process for estimating EV values for each rectangular block of video see-through image data according to the first embodiment. FIG. 22 is a diagram showing an example of the relationship between the brightness of a video see-through display image and the EV value in a scene according to the first embodiment. FIG. 23 is a diagram showing an example of the relationship between the brightness of the video see-through display image according to the first embodiment and the luminance difference of the EV value in the scene. FIG. 24 is a diagram showing an example of the relationship between the brightness of the video see-through display image and the average value of luminance for each block according to the first embodiment. FIG. 25 is a diagram showing an example of the brightness adjustment process for video see-through image data according to the first embodiment. FIG. 26 is a diagram showing an example of video see-through image data before and after correction according to the first embodiment. FIG. 27 is a diagram illustrating an example of a light-blocking target region according to the first modification of the first embodiment. FIG. 28 is a diagram illustrating an example of a case where a bright area exists according to the second modification of the first embodiment. FIG. 29 is a diagram illustrating an example of a light-blocking target region according to the second modification of the first embodiment. FIG. 30 is a diagram illustrating an example of the overall configuration of a head-mounted display according to the second embodiment. Figure 31 is a diagram showing an example of the positional relationship between the eye tracking camera and the user's eyes in the second embodiment. Figure 32 is a diagram showing another example of the positional relationship between the eye tracking camera and the user's eyes in the second embodiment. FIG. 33 is a diagram illustrating an example of the configuration of an SoC according to the second embodiment. FIG. 34 is a flowchart showing an example of the flow of processing for acquiring the correction amount of pupil position misalignment according to the second embodiment. FIG. 35 is a flowchart showing an example of the flow of a process for generating display data when using the head-mounted display according to the second embodiment.

Embodiments of the semiconductor device, method, and head-mounted display disclosed herein will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the disclosed technology. Furthermore, the embodiments can be combined as appropriate to the extent that the processing content is not contradictory.

(First embodiment)
1 is a diagram showing an example of the overall configuration of a head-mounted display 1a according to the first embodiment. The head-mounted display 1a is equipped with high-sensitivity cameras (video see-through cameras 41a and 41b) that can capture images even in dark places, and supports the user's vision by displaying camera images in dark places.

Specifically, the head-mounted display 1a includes, for example, a pair of eyeglasses 10, lenses 2a and 2b, transparent displays 3a and 3b, video see-through cameras 41a and 41b, display projectors 5a and 5b, an ambient light sensor 60, head tracking cameras 63a and 63b, and a system on a chip (SoC) 100a.

The lenses 2a, 2b, transmissive displays 3a, 3b, video see-through cameras 41a, 41b, display projectors 5a, 5b, ambient light sensor 60, head tracking cameras 63a, 63b, and SoC are fixed to the eyeglass body 10.

The eyeglass body 10 is shaped so that it can be worn on the user's head, similar to the frame of regular eyeglasses. The eyeglass body 10 includes, for example, a front portion that secures the lenses 2a and 2b, and temple portions that can be fastened to the user's earlobes.

Lens 2a and 2b are transparent eyeglass lenses positioned in front of the eyes of a user wearing head-mounted display 1a. Lenses 2a and 2b also function as liquid crystal shutters. This liquid crystal shutter function allows lenses 2a and 2b to adjust the amount of external light that passes through lenses 2a and 2b. Hereinafter, when there is no need to distinguish between individual lenses 2a and 2b, they will simply be referred to as lenses 2.

Transparent displays 3a and 3b are provided on lenses 2a and 2b and are capable of displaying images. Furthermore, transparent displays 3a and 3b transmit external light. Therefore, a user wearing head-mounted display 1a can see both the transmitted image transmitted through transparent displays 3a and 3b and the display image displayed on transparent displays 3a and 3b. The display images displayed on transparent displays 3a and 3b are, for example, images obtained by subjecting captured image data captured by video see-through cameras 41a and 41b (described below) to processing such as deformation. Note that deformation is an example of correction.

The transparent displays 3a and 3b are provided in at least a portion of the area of the lenses 2a and 2b. In this embodiment, the transparent displays 3a and 3b are provided in a portion of the area located at the center of the lenses 2a and 2b. The transparent displays 3a and 3b are, for example, screens for optical see-through displays such as waveguides. Hereinafter, when there is no need to distinguish between the individual transparent displays 3a and 3b, they will simply be referred to as the transparent displays 3.

Of the two surfaces of the transparent display 3, the surface facing the user wearing the head-mounted display 1a is referred to as the front surface. Furthermore, of the two surfaces of the transparent display 3, the surface facing away from the user wearing the head-mounted display 1a is referred to as the back surface. The user wearing the head-mounted display 1a is an example of an observer of the transparent display 3.

The video see-through cameras 41a and 41b are cameras that capture images in the direction in which the back of the transmissive display 3 faces. The direction in which the back of the transmissive display 3 faces corresponds to the front direction for a user wearing the head-mounted display 1a. More specifically, the video see-through cameras 41a and 41b are provided, for example, in positions above the centers of the lenses 2a and 2b of the eyeglass body 10. The video see-through cameras 41a and 41b are an example of a first camera in this embodiment. Furthermore, the captured image data captured by the video see-through cameras 41a and 41b is an example of first image data in this embodiment.

Video see-through cameras 41a and 41b are image sensors capable of capturing images in dark places, such as a SPAD (Single Photon Avalanche Diode) sensor or a highly sensitive CMOS (Complementary Metal Oxide Semiconductor) sensor. Alternatively, video see-through cameras 41a and 41b may be IR (Infrared Rays) sensors capable of capturing images in the dark by irradiating them with IR light. Hereinafter, when there is no need to distinguish between the individual video see-through cameras 41a and 41b, they will simply be referred to as video see-through cameras 41.

In this embodiment, the captured image data captured by the video see-through camera 41 is referred to as video see-through image data. Furthermore, when the video see-through image data is projected onto the transmissive display 3 by the display projector 5, the image displayed on the transmissive display 3 is referred to as a video see-through display image. Note that "when video see-through image data is projected onto the transmissive display 3 by the display projector 5" also includes when video see-through image data after various corrections is projected onto the transmissive display 3.

Display projectors 5a and 5b display images on transmissive displays 3a and 3b under the control of SoC 100a (described below). Display projectors 5a and 5b are, for example, micro LEDs (Light Emitting Diodes). Hereinafter, when there is no need to distinguish between individual display projectors 5a and 5b, they will simply be referred to as display projectors 5.

The Ambient Light Sensor 60, also known as an ambient light sensor, acquires ambient light information. For example, the Ambient Light Sensor 60 measures the intensity of ambient light.

The head tracking cameras 63a and 63b detect the movement of the head of a user wearing the head-mounted display 1a. Hereinafter, when there is no need to distinguish between the individual head tracking cameras 63a and 63b, they will simply be referred to as head tracking cameras 63.

SoC100a is a computer that controls each component of head-mounted display 1a. SoC100a is an example of a semiconductor device in this embodiment.

The head-mounted display 1a may further include a depth sensor that measures distance, an IMU (Inertial Measurement Unit) that measures the posture of the user wearing the head-mounted display 1a, and the like. Examples of depth sensors that measure distance include, but are not limited to, a ToF (Time of Flight) sensor or a stereo camera.

Here, the configuration of the SoC 100a will be described in detail. Figure 2 is a diagram showing an example of the configuration of the SoC 100a according to the first embodiment. As shown in Figure 2, the SoC 100a includes an I2C (Inter-Integrated Circuit) interface 11, 22, a Mono ISP (Image Signal Processor) 12, a DSP (Digital Signal Processor) & AI (Artificial Intelligence) Accelerator 13, and an SRAM (Static Random Access Memory) 1. 4a-14c, GPU (Graphics Processing Unit) 15, Color ISP 16, Time Warp 17, Warp 18, Display Controller 19, STAT (Statistics block) 21, CPU (Central Processing Unit) 23, and DRAM (Dynamic Random Access Memory) Controller 24.

Although not shown in FIG. 1, the head mounted display 1a further includes an IMU 61, a ToF sensor 62, a flash memory 31, and a DRAM 32 outside the SoC 100a. The flash memory 31 and the DRAM 32 store various types of data under the control of the SoC 100a.

Furthermore, the calibration cameras 42a and 42b shown in FIG. 2 are cameras provided outside the head-mounted display 1a. Hereinafter, when there is no need to distinguish between the individual calibration cameras 42a and 42b, they will simply be referred to as calibration cameras 42. The calibration camera 42 is an example of the second camera in this embodiment. Details of the calibration camera 42 will be described later with reference to FIG. 4.

I2C interfaces 11 and 22 are communication interfaces that perform synchronous serial communication. I2C interface 11 acquires acceleration and angular velocity from IMU 61. I2C interface 22 acquires the intensity of ambient light from Ambient Light Sensor 60.

Mono ISP 12 acquires information from various sensors used to recognize the surrounding situation and corrects the acquired information. For example, Mono ISP 12 acquires distance measurement data from surrounding objects from ToF sensor 62. Mono ISP 12 also acquires image data from head tracking cameras 63a and 63b, for example, and corrects brightness, etc.

The DSP & AI Accelerator 13 includes a DSP 131 and an AI Accelerator 132. The DSP & AI Accelerator 13 performs various tracking processes based on various data acquired and corrected by the Mono ISP 12. For example, the DSP & AI Accelerator 13 may perform head tracking processing to detect head movement of a user wearing the head-mounted display 1a. Alternatively, the DSP & AI Accelerator 13 may perform hand tracking processing or eye tracking processing, depending on the type of sensor. The DSP & AI Accelerator 13 outputs tracking information generated as a result of various tracking processes to the Time Warp 17. Note that various tracking processes are not required in this embodiment.

SRAMs 14a to 14c store various data used in various processes and data generated by various processes. For example, SRAM 14b is used as a temporary storage location for rendered image data and captured image data. It may also store first positional relationship data used in the image data transformation process performed by Warp 18, described below. SRAM capacity is usually small and may not be able to store the entire image. In such cases, SRAM may be used as a FIFO (First in First out) buffer to store only a portion of the image that has not yet been used in subsequent processing. SRAMs 14a to 14c are an example of memory in this embodiment. Note that each process performed by SoC 100a uses SRAMs 14a to 14c as a buffer for temporary data storage. While DRAM 32 can also be used as a buffer, using SRAMs 14a to 14c reduces power consumption. Note that the first positional relationship data stored in SRAM 14b uses data previously stored in flash memory 31. For this reason, flash memory 31 is included as an example of memory in this embodiment. Flash memory 31 is an external storage device, and may be NAND memory, an SSD, an SD card, or the like.

The GPU 15 performs rendering processing of the image data displayed on the transmissive display 3.

Time Warp 17 is a processing circuit that corrects delays in image data caused by processing by GPU 15. The time correction performed by Time Warp 17 has the effect of reducing the discomfort experienced by users who view image data displayed on the translucent display 3, known as "VR (Virtual Reality) sickness." Time Warp 17 performs correction processing using, for example, the results of head tracking processing performed by DSP & AI Accelerator 13.

Color ISP 16 and Warp 18 acquire captured image data from the video see-through camera 41 and the calibration camera 42. Color ISP 16 converts the acquired image data into RGB (Red-Green-Blue color model) image data.

Warp 18 performs correction processing on the captured image data of the video see-through camera 41 that has been converted into RGB image data by Color ISP 16. The correction processing by Warp 18 includes, for example, a transformation processing based on the first positional relationship data for at least a partial area of the captured image data, and a processing for extracting at least a partial area of the captured image data as a display target. Through this transformation and extraction processing, Warp 18 generates video see-through image data so that the outline of the subject included in the transmitted image and the outline of the subject included in part of the captured image data are continuous on the transmissive display 3. The video see-through image data is an example of display data in this embodiment. Warp 18 is an example of a display data generation circuit in this embodiment.

STAT21 identifies various pieces of information from the image data captured by the video see-through camera 41, which has been converted into an RGB image by Color ISP16. For example, STAT21 detects the information required for AE (Auto Exposure) and AWB (Auto White Balance). STAT21 also extracts a histogram from the captured image data, calculates the average brightness value for each rectangular block of the captured image data, and counts the number of saturated pixels. A histogram is a graph that shows the distribution of brightness values of pixels in image data, and typically the horizontal axis represents brightness and the vertical axis represents the number of pixels.

By performing processing by Color ISP 16, Warp 18, and STAT 21 with low latency, it is possible to reduce VR sickness experienced by a user wearing the head-mounted display 1a.

The CPU 23 performs AE and AWB based on information acquired via the STAT 21 and the I2C interface 22. The CPU 23 also performs extraction processing of bright and dark areas based on the brightness of the scene and the luminance of the captured image data. Based on the results of these processing, the CPU 23 generates transmittance data for controlling the liquid crystal shutter of the lens 2a. Details of the extraction processing of bright and dark areas by the CPU 23 will be described later. The CPU 23 is an example of an image processing circuit in this embodiment.

The display controller 19 controls the display projector 5 to display an image on the transmissive display 3. The display controller 19 also controls the degree of transmittance of the liquid crystal shutter of the lens 2 based on transmittance data generated by the CPU 23. The display controller 19 also corrects the video see-through image data corrected by the Warp 18 and the rendering image data rendered by the GPU 15 into a format suitable for display on the transmissive display 3. The display controller 19 is an example of a display control circuit in this embodiment.

More specifically, the Display Controller 19 comprises an EN (Enable) block 191 and a Blend block 192. The EN block 191 determines whether the display of saturated pixels is on or off. If the EN block 191 determines that the pixel is off, it corrects the pixel to be hidden (black). If the EN block 191 determines that the pixel is on, it corrects the brightness of the displayed image corresponding to the pixel. The Blend block 192 combines the rendering image data and the video see-through image data.

The DRAM controller 24 controls the storage of various data in the DRAM 32 and the reading of various data from the DRAM 32.

Note that the configuration of the head-mounted display 1a in this embodiment is not limited to the example shown in Figures 1 and 2. For example, the head-mounted display 1a does not necessarily include the head tracking camera 63a, IMU 61, and ToF sensor 62. Furthermore, the head-mounted display 1a may also include other sensors, etc.

Next, we will explain the calibration process performed before using the head-mounted display 1a. The calibration process is performed, for example, before the head-mounted display 1a is shipped. More specifically, the calibration process is a process for making corrections that take into account differences between the eye position of a user wearing the head-mounted display 1a and the mounting position of the video see-through camera 41.

When a user wears the head-mounted display 1a, the user's eyes are generally positioned near the centers of the lenses 2a and 2b, respectively. Therefore, the transmitted image that passes through the lenses 2a and 2b and the transmissive displays 3a and 3b is based on a viewpoint near the centers of the lenses 2a and 2b. However, it is structurally difficult to mount the video see-through cameras 41a and 41b so that they are aligned with the optical axis of the user's eyes. For this reason, as shown in Figure 1, the video see-through cameras 41a and 41b are provided in the eyeglass body 10 of the head-mounted display 1a, rather than near the centers of the lenses 2a and 2b.

For these reasons, there is a difference between the viewpoint of the video see-through image data captured by the video see-through cameras 41a and 41b and the viewpoint of the user wearing the head-mounted display 1a. Differences in the angle of view between the transparent display 3 and the video see-through camera 41, as well as distortion of the image due to lens distortion, also contribute to the misalignment. Note that lens distortion is caused by both the characteristics of the lens of the video see-through camera 41 and distortion of the lens 2 of the head-mounted display 1a.

FIG. 3 is a diagram showing an example of the difference between the video see-through display images 931, 931a and the transmission image 941 according to the first embodiment. The video see-through display image 931 is an image in which the uncorrected video see-through image data is displayed as is on the transmission display 3. In this case, as shown in FIG. 3, a difference occurs in the position and/or size of the object (subject 9a) included in the video see-through display image 931 and the object (subject 9a) included in the transmission image.

In Figure 3, the video see-through display image 931 and the transmission image 941 are displayed separately, but in reality, the video see-through display image 931 is displayed superimposed on the transmission image 941 on the transmission display 3. In this case, the contours of the subject 9a in the video see-through display image 931 and the subject 9a in the transmission image 941 are not continuous, resulting in a double image. As a result, a user wearing the head-mounted display 1a sees both the subject 9a in the video see-through display image 931 and the subject 9a in the transmission image 941 as a double image, reducing visibility. Note that in this embodiment, the "subject" includes not only objects depicted in the captured image data, but also objects included in the transmission image. Note that objects also include people.

For this reason, when the head-mounted display 1a is in use, the SoC 100a corrects the video see-through image data before displaying it on the transmissive display 3. The position and size of the subject 9a in the video see-through display image 931a based on the corrected video see-through image data match the position and size of the subject 9a in the transmissive image 941. In this case, the subject 9a does not appear as a double image on the transmissive display 3, so user visibility is not reduced. The calibration process acquires first positional relationship data used for such correction during use. The first positional relationship data is data indicating the positional relationship between the subject 9a included in the transmissive image 941 transmitted through the transmissive display 3 and the subject included in the first image data captured by the video see-through camera 41. For example, the first positional relationship data represents the amount of alignment correction of the first image data in the warp process. The alignment correction amount is defined, for example, by coordinates that represent the positional relationship of feature points before and after the warp process.

FIG. 4 is a diagram showing an example of the installation positions of the calibration cameras 42a and 42b according to the first embodiment. As shown in FIG. 4, the calibration cameras 42a and 42b are installed at positions corresponding to the eye positions of a user wearing the head-mounted display 1a. For example, as shown in FIG. 4, the calibration cameras 42a and 42b are installed near the centers of the lenses 2a and 2b. At these positions, the calibration cameras 42a and 42b pass through the transparent display 3 from the front side of the transparent display 3 and capture an image in the direction in which the rear side of the transparent display 3 faces. Therefore, the calibration cameras 42a and 42b pass through the transparent display 3 to capture an image of a subject located on the rear side of the transparent display 3. Note that the calibration camera 42 is not used after the calibration process and is therefore not included in the configuration of the head-mounted display 1a.

It is desirable for the subject used for calibration processing to be one from which feature points can be easily extracted.

FIG. 5 is a diagram showing an example of a subject 9b for calibration processing according to the first embodiment. In this embodiment, as shown in FIG. 5, a checkerboard chart in which black and white rectangles are arranged in a grid pattern is used as the subject 9b for calibration processing. The grid points of the checkerboard chart are examples of feature points 90. Note that while FIG. 5 shows one grid point as an example of a feature point 90, all grid points in the image data captured by the calibration camera 42 and the video see-through camera 41 become feature points 90. Note that the subject 9b for calibration processing is not limited to the example shown in FIG. 5.

FIG. 6 is a diagram showing an example of a side view of subject 9b photographed by the calibration camera 42 according to the first embodiment. As shown in FIG. 6, both the video see-through camera 41 and the calibration camera 42 photograph subject 9b located toward the back surface 302 of the transmissive display 3. The calibration camera 42 photographs subject 9b from the front surface 301 of the transmissive display 3 toward the back surface 302, passing through the lens 2 and the transmissive display 3. The video see-through camera 41 photographs subject 9b without passing through the lens 2 or the transmissive display 3.

Here, we will explain the flow of the calibration process using the calibration camera 42 described in Figures 4 to 6.

FIG. 7 is a flowchart showing an example of the flow of calibration processing according to the first embodiment. FIG. 8 is a diagram showing an example of the processing content of each step in the flowchart of FIG. 7. Steps 1-7 in FIG. 8 correspond to S1-S7 in FIG. 7.

First, Color ISP 16 acquires image data (first calibration image data 901 shown in FIG. 8) of subject 9b photographed by calibration camera 42 (S1). The first calibration image data 901 is image data of subject 9b photographed from the front surface 301 of the transparent display 3, passing through the transparent display 3 to the rear surface 302 of the transparent display 3. The first calibration image data 901 is an example of second image data in this embodiment. Note that calibration camera 42 does not have to be mounted on a head-mounted display, and a commercially available digital camera, web camera, etc. may be used. In this case, the photographed image data is transferred from the digital camera, web camera, etc. to SoC 100a after photographing, and subsequent processing is performed.

Then, the CPU 23 extracts feature points 90 of the subject 9b in the first calibration image data 901 (S2). The feature points 90 extracted from the first calibration image data 901 are designated as first feature points. Extracting feature points 90 means identifying the positions of the feature points 90 (for example, grid points of a checkerboard chart) depicted in the first calibration image data 901.

Here, the first calibration image data 901 is a pseudo-image of the transmission image seen by a user wearing the head-mounted display 1a, but differs from the transmission image actually seen by the user due to lens distortion of the calibration camera 42, etc. Distortion caused by the characteristics of the calibration camera 42, including lens distortion, is called camera distortion. To remove the camera distortion, the CPU 23 corrects the first calibration image data 901 using the internal parameters of the calibration camera 42 before extracting the first feature point in S2. Correction using the internal parameters is an inverse correction that cancels out distortion caused by the display system. Through this processing, the CPU 23 converts the first calibration image data 901 into a state equivalent to the transmission image seen by a user wearing the head-mounted display 1a. In Figure 8, the converted image data is shown as pseudo transmission image data 911.

The internal parameters of the calibration camera 42 are correction parameters for removing the effects of camera distortion of the calibration camera 42. The internal parameters can be calculated using a known calibration tool such as OpenCV (registered trademark). The internal parameters may be calculated by an information processing device external to the SoC 100a before the processing of Figure 7 and stored in memory within the SoC 100a. Furthermore, the video see-through camera 41, like the calibration camera 42, also has camera distortion. Therefore, the internal parameters for removing the effects of camera distortion of the video see-through camera 41 may also be stored in memory within the SoC 100a.

Next, Color ISP 16 acquires image data (video see-through image data 921 shown in Figure 8) of subject 9b captured by video see-through camera 41 from video see-through camera 41 (S3).

Then, the display controller 19 controls the display projector 5 to display the video see-through image data 921 on the transparent display 3 (S4). In Figure 8, the video see-through image data 921 displayed on the transparent display 3 is shown as a video see-through display image 932. Furthermore, after the processing of S4, the subject 9b is removed from in front of the head-mounted display 1a by an operator or the like. As a result, the subject 9b is no longer within the angle of view of the calibration camera 42.

Next, Color ISP 16 acquires image data from the calibration camera 42, which is an image of the transmissive display 3 displaying the video see-through display image 932 (S5). This image data is the second calibration image data 902 shown in FIG. 8. The second calibration image data 902 is also an example of the third image data in this embodiment.

Then, the CPU 23 extracts feature points 90 of the subject 9b in the second calibration image data 902 (S6). The feature points 90 extracted from the second calibration image data 902 are designated as second feature points. More specifically, similar to the processing in S2, the CPU 23 corrects camera distortion of the second calibration image data 902 using internal parameters of the calibration camera 42 before extracting the second feature points. The corrected second calibration image data 902 is a pseudo-reproduction of the video see-through display image seen by a user wearing the head-mounted display 1a, and is therefore referred to as pseudo see-through display image data 940. The CPU 23 extracts the second feature points from the pseudo see-through display image data 940.

The CPU 23 then calculates deformation parameters from the first feature points extracted in S2 and the second feature points extracted in S6 (S7). The deformation parameters are the first positional relationship data described above. The deformation parameters are data indicating the positional relationship between the subject 9b included in the pseudo-transmitted image data 911 and the subject 9b included in the video see-through image data 921 captured by the video see-through camera 41. The pseudo-transmitted image data 911 in the calibration process corresponds to the transmitted image when the head-mounted display 1a is actually in use. Furthermore, the pseudo-see-through display image data 940 generated from the video see-through image data 921 in the calibration process corresponds to the video see-through display image when the head-mounted display 1a is actually in use. Therefore, the deformation parameters calculated in S7 indicate the positional relationship between the subject included in the transmitted image and the subject included in the video see-through image data captured by the video see-through camera 41.

More specifically, in the calculation process of the deformation parameters (first positional relationship data) in S7, the CPU 23 calculates the positional relationship between the first feature point and the second feature point, thereby calculating deformation parameters that can match the first feature point and the second feature point. As shown in Step 7-1 of FIG. 8, the CPU 23 calculates deformation parameters that can correct the pseudo see-through display image data 940 so that it matches the pseudo transmitted image data 911. Also, as shown in Step 7-2 of FIG. 8, the CPU 23 calculates the amount of distortion caused by the display system based on the feature point 90 of the subject 9b in the video see-through image data 921 and the feature point 90 of the subject 9b in the pseudo see-through display image data 940. The amount of distortion caused by the display system is the magnitude of the image distortion caused by the characteristics of the lens 2 and the transparent display 3. The characteristics of the lens 2 and the transparent display 3 include, for example, the curvature of the lens 2 and the transparent display 3. The CPU 23 calculates the deformation parameters (first positional relationship data) based on the deformation parameters calculated in Step 7-1 and the distortion amount calculated in Step 7-2. More specifically, the CPU 23 performs a conversion process on the deformation parameters calculated in Step 7-1. This conversion process will be described later with reference to Figures 9-11.

Then, the CPU 23 stores the calculated transformation parameters (first positional relationship data) in a memory such as the SRAM 14b or the flash memory 31 (S8). At this point, the processing of this flowchart ends.

The calibration process shown in FIG. 7 may be executed by the SoC 100a as described above, or may be executed by another information processing device external to the head-mounted display 1a. The other information processing device may be, for example, a high-performance PC (Personal Computer). In addition, the first positional relationship data generated by the calibration process executed by the SoC 100a of one head-mounted display 1a may be stored in the memory of multiple head-mounted displays 1a.

Here, the calculation process for the transformation parameters (first positional relationship data) described in Figures 7 and 8 will be explained in more detail using Figures 9-11.

FIG. 9 is a diagram showing an example of distortion contained in each image data in the calibration process according to the first embodiment. The deformation parameter that can be directly calculated in Step 7-1 shown in FIG. 8 above is deformation parameter A for the pseudo see-through display image data 940 based on the second calibration image data 902 captured by the calibration camera 42.

As shown in FIG. 9, the video see-through image data 921 acquired in S3 of FIG. 7 is affected by distortion due to the display system when it is displayed on the transmissive display 3 in Step 4. For this reason, the second calibration image data 902 acquired in Step 5 also includes distortion due to the display system. The second calibration image data 902 also includes distortion (camera distortion) due to the calibration camera 42. In Step 6, the CPU 23 removes this camera distortion through correction using internal parameters. Then, in Step 7-1, the CPU 23 calculates a deformation parameter A that can correct the pseudo see-through display image data 940 from which the camera distortion has been removed so that it matches the pseudo transmission image data 911. The pseudo see-through display image data 940a after deformation using the deformation parameter A matches the pseudo transmission image data 911.

However, when the head-mounted display 1a is in use, the calibration camera 42 is not used, and therefore the object to be deformed during use is the video see-through image data 921.

FIG. 10 is a diagram showing an example of distortion contained in each image data when the head-mounted display 1a according to the first embodiment is in use. Also, FIG. 11 is a diagram showing an example of the breakdown of deformation process B in FIG. 10.

As shown in Figure 10, the correction process when using the head-mounted display 1a aims to ensure that the feature points of the subject in the video see-through display image 933 displayed on the transmissive display 3 match the feature points of the subject in the transmissive image.

For this reason, deformation process B when using the head-mounted display 1a requires deformation parameters B that take into account the distortion that occurs in the video see-through image data 921a after deformation.

Specifically, when the video see-through image data 921 is displayed on the transmissive display 3, it is affected by distortion due to the display system and camera distortion of the video see-through camera 41. The effect of camera distortion is removed by correction using internal parameters, so it does not need to be taken into account in the transformation process.

As a result, as shown in FIG. 11, transformation process B is a combination process of transformation process S701 for distortion caused by the display system, transformation process S702 using transformation parameter A calculated in the calibration process, and inverse transformation process S703 for distortion caused by the display system. The CPU 23 calculates transformation parameter B by combining the transformation parameters from processes S701 to S703. Transformation parameter B indicates, for example, the difference in the positions of feature points 90 (e.g., lattice points) of subject 9b before and after processes S701 to S703. This transformation parameter B is the first positional relationship data.

Next, we will explain the processing that occurs when using the head-mounted display 1a.

FIG. 12 is a flowchart showing an example of the flow of the display data generation process when using the head-mounted display 1a according to the first embodiment. Before the process in FIG. 12, it is assumed that the calibration process in FIG. 7 has been completed and the transformation parameters (first positional relationship data) have already been stored in a memory such as SRAM 14b or flash memory 31.

First, Color ISP 16 acquires video see-through image data from video see-through camera 41 (S21). Color ISP 16 then converts the acquired video see-through image data into RGB image data. STAT 21 extracts a histogram, calculates the average brightness value for each rectangular block, and counts the number of saturated pixels for the video see-through image data converted into an RGB image by Color ISP 16. Color ISP 16 and STAT 21 store the video see-through image data converted into RGB image data, the histogram, the calculation results of the average brightness value for each rectangular block, and the number of saturated pixels in memory such as SRAM 14a-14c or DRAM 32. Hereinafter, unless a distinction is needed, video see-through image data converted into RGB image data will also be simply referred to as video see-through image data.

Next, Warp 18 acquires the transformation parameters (first positional relationship data) generated in the calibration process of FIG. 7 from SRAMs 14a to 14c, etc. (S22).

The CPU 23 also performs a process of extracting bright and dark areas from the video see-through image data converted into RGB image data (S23). For example, the CPU 23 extracts bright areas from the video see-through image data, and calculates the bright areas of the transmitted image based on the bright areas from the video see-through image data and the first positional relationship data. Details of the process of extracting bright and dark areas will be described later.

Furthermore, Warp 18 deforms at least a portion of the video see-through image data according to the alignment correction amount defined by the first positional relationship data (S24). This deformation makes it possible to align the contour of the subject included in the transmitted image with the contour of the subject included in the video see-through display image. Therefore, even when the transmitted image is visible, it is possible to prevent the contour of the subject included in the transmitted image and the contour of the subject included in the video see-through display image from appearing double.

As mentioned above, the head-mounted display 1a displays a video see-through image to aid vision in dark places. For this reason, not all areas of the video see-through image data are necessarily subject to display. For this reason, Warp 18 corrects only the areas that require display, based on the results of the bright and dark area extraction process in S23.

FIG. 13 is a diagram showing an example of correction of video see-through image data 922 according to the first embodiment. As shown in FIG. 13, Warp 18 deforms only areas of the video see-through image data 922 that have been determined to be dark or dimly lit in the process of extracting bright and dark areas. Warp 18 does not deform areas determined to be bright. For example, Warp 18 deforms the parts of the video see-through image data 922 that are dark or dimly lit and extracted as display targets, and that border the bright areas of the transmitted image, to generate display data.

Warp 18 sets the value "0" to areas not subject to correction. In Figure 13, the areas at both ends of the video see-through image data 922 are determined to be bright areas, so Warp 18 sets the value "0" to those areas. In areas where the value "0" is set, the original image is deleted. In areas of the corrected video see-through image data 922a where the value "0" is set, nothing is displayed when displayed on the transmissive display 3. The areas shown in black in the corrected video see-through image data 922a in Figure 13 have the value "0" set, so when actually displayed on the transmissive display 3, the background is visible through them. Therefore, in areas where the value "0" is set, the user only sees the transmissive image. By Warp 18 narrowing down the areas subject to correction in this way, the amount of calculation required for correction can be reduced. Warp 18 outputs the corrected video see-through image data 922a to the Display Controller 19.

Returning to FIG. 12, the Display Controller 19 adjusts the brightness of a portion of the corrected video see-through image data 922a based on the results of the bright and dark area extraction process of S23 (S25). The Display Controller 19 performs, for example, gamma correction to correct the brightness. The Display Controller 19 may also perform color correction to adjust the colors.

Then, the Display Controller 19 generates display data by converting the corrected video see-through image data 922a into a format that can be displayed on the transmissive display 3 (S26). The processing content differs depending on the transmissive display 3, but the Display Controller 19 performs, for example, resizing to match the resolution.

Then, the Display Controller 19 outputs the generated display data and the transmittance data generated in the bright and dark area extraction process of S23 (S27). More specifically, the generated display data is output to the display projector 5, causing the transmissive display 3 to display a video see-through display image. The Display Controller 19 also controls the transmittance of the liquid crystal shutter of the lens 2 based on the transmittance data. For example, the Display Controller 19 controls the transmittance of the area of the transmissive display 3 corresponding to the bright area of the transmitted image to a second transmittance that is smaller than the first transmittance used when the bright area of the transmitted image was calculated.

Here, the processing of this flowchart ends. The processing of the flowchart in Figure 12 is repeatedly executed while the head-mounted display 1a is being used by the user. Note that, in order to reduce the user's visual discomfort, it is desirable that the processing of Figure 12 be executed at a refresh rate of 90 to 60 Hz, as an example.

FIG. 14 is a diagram illustrating the principle of visual assistance using the transmissive display 3 and liquid crystal shutter according to the first embodiment. FIG. 14 shows a state in which a video see-through display image is displayed on the transmissive display 3 and light reduction is performed by the liquid crystal shutter of the lens 2 as a result of the processing of S27 in FIG. 12. The lens 2 with liquid crystal shutter function and the transmissive display 3 are located between the eye 8 of the user wearing the head-mounted display 1a and the real image (subject).

For example, in Figure 14, there are dark areas (dark regions) 71 and bright areas (bright regions) 72 in the real image. The Display Controller 19 displays a video see-through display image in the areas where the dark areas of the real image on the transmissive display 3 are transmitted. Therefore, in the areas where the dark areas 71 are transmitted, the user sees a bright video see-through display image 934, not a dark transmitted image. Note that even if a small amount of the transmitted image is visible in the areas where the dark areas 71 are transmitted, no double image occurs because the video see-through display image 934 has been corrected to match the transmitted image by the transformation process in S24.

Furthermore, in the area on the transmissive display 3 where the bright area 72 is transmitted, a value of "0" is set for each pixel of the video see-through display image 934, so the video see-through display image 934 is not displayed on the transmissive display 3. Furthermore, depending on the brightness of the bright area 72, the liquid crystal shutter function of the lens 2 dims the area where the bright area 72 is transmitted.

FIG. 15 is a diagram showing an example of the display mode of the transmissive display 3 according to the first embodiment. As shown in FIG. 15, without the visual aids of the video see-through display image 934 and the liquid crystal shutter, the user may be unable to see objects in the region 310 of the transmissive display 3 through which the dark area 71 is transmitted due to darkness. Furthermore, in the region 320 of the transmissive display 3 through which the bright area 72 is transmitted, the user may be unable to see objects due to glare.

In contrast, with the visual aids of the video see-through display image 934 and the liquid crystal shutter, the user's visibility is improved in the region 310 of the transmissive display 3 through which the dark area 71 is transmitted by the video see-through display image 934. Furthermore, in the region 320 of the transmissive display 3 through which the bright area 72 is transmitted by the liquid crystal shutter, the glare is reduced, thereby improving the user's visibility.

Next, the details of the bright and dark area extraction process in S23 of Figure 12 described above will be explained. Figure 16 is a diagram showing an example of the flow of the bright and dark area extraction process according to the first embodiment.

First, the CPU 23 acquires the brightness of the scene (EV (Exposure Value) value) (S231). Generally, in a normal camera, in order to keep the brightness of the captured image data constant, the brightness of the scene is estimated using AE and the shutter speed and sensitivity are determined. In this embodiment, the CPU 23 uses this AE information to acquire the brightness of dark areas of the scene (EV value). The brightness of the scene is the brightness around the user wearing the head-mounted display 1a.

The CPU 23 also determines the transmittance of lens 2 based on the acquired EV value (S232). The transmittance of lens 2 indicates the degree of light blocking by the liquid crystal shutter.

Then, the CPU 23 extracts bright and dark areas from the video see-through image data 923 (S233). Extracting bright and dark areas means, in other words, identifying the ranges that correspond to bright and dark areas in the video see-through image data 923. At this point, the processing of this flowchart ends.

Here, the details of the EV value acquisition process of S231 in Figure 16 will be explained. Figure 17 is a diagram showing an example of the flow of the EV value acquisition process according to the first embodiment. The processes of S301 to S307 shown in Figure 17 are executed by the CPU 23. The CPU 23 acquires the histogram and the calculation results of the average brightness value for each rectangular block generated by STAT21 from the video see-through image data 923 acquired by Color ISP16.

The CPU 23 calculates a threshold value that serves as a reference for dark places from the histogram. For example, the CPU 23 integrates the histogram from element 0 and sets the element number when the integrated value exceeds N% of the total number of pixels as the threshold value (S301). Figure 18 is a diagram showing an example of a threshold value determined based on a histogram according to the first embodiment. By calculating the threshold value from the histogram, the CPU 23 can dynamically set a threshold value that corresponds to the video see-through image data 923.

Returning to FIG. 17, the CPU 23 obtains the average value of the area where the brightness is equal to or less than the threshold value from the average brightness values of rectangular blocks obtained from the STAT 21 (S302). This average value is called the STAT average value.

Then, the CPU 23 estimates the brightness (EV value) of the scene from the STAT average value and the current sensor module settings (exposure time, sensor gain, and aperture) (S303).

The CPU 23 also sets new sensor setting values for various sensors (such as the Ambient Light Sensor 60) based on the estimated brightness (S304). This allows the sensor sensitivity to be adjusted to match the actual level of brightness.

The CPU 23 also acquires the ambient light brightness (Lux) from the Ambient Light Sensor 60 via the I2C interface 22 (S305).

Then, the CPU 23 converts Lux to an EV value based on a known EV/Lux conversion table (S306). Figure 19 is a diagram showing an example of a common EV/Lux conversion table. The EV value is a numerical value used when determining the exposure compensation value of a camera, and it is known that there is a correspondence relationship between the EV value and illuminance (Lux value), a common unit of brightness, roughly as shown in the table in Figure 19. The higher the EV value, the brighter the scene, and the lower the EV value, the darker the scene. As in the example shown in Figure 19, the CPU 23 may determine that an EV value of "-4" or greater and less than "0" is a "dark place," an EV value of "0" or greater and less than "2" is a "twilight" place, and an EV value of "15" or greater is a "bright place." Note that the criteria for "dark place," "twilight," and "bright place" shown in Figure 19 are merely examples and are not limited to these. Also, in Figure 19, EV values of "2" or greater and less than "15" are considered "daytime" and distinguished from "bright places," but "daytime" may also be included in "bright places."

Returning to FIG. 17, if the ambient light brightness measurement results from the Ambient Light Sensor 60 are available, the CPU 23 uses the detection results (ALS brightness) of the Ambient Light Sensor 60; if the measurement results are unavailable, the CPU 23 uses the AE information (AE brightness) (S307). When the measurement results from the Ambient Light Sensor 60 are used, the update interval is longer than when AE information is used, but power consumption for scene brightness estimation processing can be reduced. Furthermore, since the sensor settings are set so that dark areas can be captured brightly, bright areas are saturated, and AE information based on the captured image data may not accurately measure the brightness of bright areas. Using the ALS detection results, which include the degree of brightness in bright areas, makes it possible to obtain more accurate information about the brightness of the surrounding environment. The CPU 23 outputs an EV value based on either the ALS brightness or the AE brightness to the Display Controller 19. The CPU 23 may use both the ALS brightness and the AE brightness to estimate the brightness of a scene.

When using AE information to estimate the brightness of a scene, the CPU 23 corrects the EV value based on the maximum value of the histogram and the average STAT value to obtain the brightness of a bright place. Specifically, the CPU 23 corrects the EV value based on the AE information using the following formula (1).

Here, the process of determining the transmittance of lens 2 in S232 of FIG. 16 will be described in detail. The CPU 23 determines the transmittance of lens 2 according to the EV value indicating the brightness of the scene obtained in S231.

When the EV value is brighter than a certain threshold, it becomes very bright and visually difficult for the user. For this reason, the CPU 23 reduces the decrease in visibility due to brightness by controlling the transmittance of the lens 2 according to the degree of brightness. For every 1 increase in the EV value, the brightness (Lux) approximately doubles. For this reason, for example, the CPU 23 may define an EV value of 15 or higher as a bright place, as shown in Figure 19, and double the degree of light blocking by the lens 2 for every 1 increase in the EV value from 15. By controlling the liquid crystal shutter in this way, the brightness of the transmitted image through the lens 2 will not exceed an EV value of 14, reducing the glare felt by the user.

FIG. 20 is a diagram showing an example of the relationship between the EV value and the transmittance of the lens 2 according to the first embodiment. As an example, the transmittance changes as shown in the graph in FIG. 20. Normally, the maximum transmittance of the lens 2 cannot be set to 100%, so in FIG. 20, the maximum transmittance is set to 80% as an example. In the example shown in FIG. 20, the transmittance is halved every time the EV value increases by 1 (brightness doubles). The CPU 23 sets the degree of light blocking of the liquid crystal shutter based on the determined transmittance.

Here, the details of the bright and dark area extraction process S233 in Figure 16 will be explained. Figure 21 is a diagram showing an example of the EV value estimation process for rectangular blocks of video see-through image data 923 according to the first embodiment.

The CPU 23 estimates the brightness for each location on the video see-through image data 923 to determine areas where visual correction is required. This brightness estimation process can be performed on a pixel-by-pixel basis, but the amount of calculation can be reduced by estimating the brightness on a block-by-block basis using the average value for each block obtained from STAT 21. Note that the various numerical values shown in Figure 21 are merely examples and are not limited to these.

The CPU 23 converts the average value for each block into an EV value.

Specifically, the CPU 23 estimates the brightness (EV value) for each block using the following equation (2):

Furthermore, although the video see-through image data 923 is an image captured without using the lens 2, the brightness of the scene actually seen by the user's eye 8 is the brightness transmitted through the lens 2. For this reason, the CPU 23 corrects the EV value of each block based on the transmittance of the lens 2 at the time of processing, using the following equation (3). Through this correction, the CPU 23 estimates the EV value indicating the brightness of the actual transmitted image. The EV value after correction based on the transmittance of the lens 2 is also called the visual EV value.

For example, the CPU 23 may divide the area into "bright areas" that are visually recognizable to the user and "dark areas" that are difficult to recognize, depending on the brightness (visual EV value). The CPU 23 may also divide the area into "twilight areas" that are brighter than dark areas but somewhat difficult to recognize.

Here, there are two patterns where darkness or brightness makes visual recognition difficult: the first pattern is when "it is simply too dark, making recognition difficult," and the second pattern is when "it is difficult to recognize due to a large difference in brightness." In the first pattern, the CPU 23 can identify the relevant area using the absolute value of the EV value. Regarding the second pattern, the dynamic range of human vision is said to be around 80 to 120 dB, and if there is a brightness difference greater than this, it becomes difficult to see dark areas. For this reason, in order to include these two patterns, the CPU 23 determines that areas where the absolute value of the EV value is below a certain threshold, or where the difference from the maximum EV value is below a certain value, are difficult to recognize.

Specifically, in the case of the first pattern, for example, the CPU 23 determines that an EV value in the range of "0" to "1" is "twilight," and as the level of visual recognition is high, gradually begins providing visual assistance using video see-through image data. Furthermore, if the EV value falls below "0," the CPU 23 determines that the location is a "dark place" and provides visual assistance using video see-through image data.

FIG. 22 is a diagram showing an example of the relationship between the brightness of the video see-through display image and the EV value in a scene according to the first embodiment. The vertical axis of FIG. 22 is the brightness (luminance) of the video see-through display image, and the horizontal axis is the EV value. "Gradually starting visual assistance" means, for example, as shown in the graph in FIG. 22, that the brightness of the video see-through display image displayed on the transmissive display 3 gradually increases as the EV value decreases.

Furthermore, in the case of the second pattern, for example, the CPU 23 determines the brightness of the video see-through display image based on the difference between the EV value of the block and the maximum EV value for the entire video see-through image data 923. For example, the CPU 23 determines that it is "twilight" if "EV value - maximum EV value" falls below "-4." In the case of "twilight," visual recognition becomes somewhat difficult, so the CPU 23 gradually begins providing visual assistance using the video see-through image data. Furthermore, the CPU 23 determines that it is a "dark place" if "EV value - maximum EV value" falls below "-7." If the CPU 23 determines that it is a "dark place," it provides visual assistance using the video see-through image data.

FIG. 23 is a diagram showing an example of the relationship between the brightness of the video see-through display image according to the first embodiment and the luminance difference in EV value in a scene. The vertical axis of FIG. 23 is the brightness (luminance) of the video see-through display image, and the horizontal axis is the EV value. This is the difference (luminance difference) between the EV value of the block and the maximum EV value in the entire video see-through image data 923. For example, as shown in the graph in FIG. 23, the CPU 23 gradually brightens the brightness of the video see-through display image displayed on the transmissive display 3 as the luminance difference increases.

By converting the brightness of the video see-through image data 923 based on the two graphs shown in Figures 22 and 23, the CPU 23 can determine the brightness of each area corresponding to "bright place," "dark place," and "twilight" based on the EV value.

As a third pattern, the video see-through image data 923 may contain locally bright areas such as blown-out highlights in addition to the brightness of the entire scene. This is because when a normal image sensor is used in the video see-through camera 41, the dynamic range is narrow, causing blown-out highlights (a state in which pixel values are saturated) in areas with relatively high brightness. An example of an area with relatively high brightness is an area with strong backlight. If the video see-through display image displayed on the transmissive display 3 includes such blown-out highlight areas, this can actually impair vision, which is undesirable. Therefore, the CPU 23 also determines such relatively bright areas to be bright areas. Areas of the video see-through display image that the CPU 23 determines to be bright areas are not displayed on the transmissive display 3.

FIG. 24 is a diagram showing an example of the relationship between the brightness of a video see-through display image and the average luminance value for each block according to the first embodiment. The CPU 23 performs a conversion as shown in the graph in FIG. 24 on the average luminance value for each block output from STAT 21.

In response to the results of this bright and dark area extraction process by the CPU 23, the Display Controller 19 adjusts the brightness of each area of the video see-through display image. For example, the Display Controller 19 displays the video see-through display image on the transmissive display 3 only in areas that it has determined should be displayed on the transmissive display 3 in all of the three patterns where visual recognition is difficult due to darkness or brightness described above. For example, the Display Controller 19 adjusts the brightness of the video see-through image data 923 by multiplying each pixel value of the video see-through image data 923 by the brightness value of that image data.

25 is a diagram showing an example of the brightness adjustment process for the video see-through image data 923 according to the first embodiment. As shown in FIG. 25, the CPU 23 determines the "bright" and "dark" areas of the video see-through image data 924 based on the brightness (EV value) of the transmitted image estimated by the CPU 23 and the average brightness value for each block of the video see-through image data 924 output from the STAT 21. The CPU 23 also identifies areas of relatively high brightness, such as blown-out highlights, in the video see-through image data 924. The Display Controller 19 adjusts the brightness of the video see-through image data 924 based on the processing results of the CPU 23. For example, the Display Controller 19 identifies the area to be displayed in the video see-through image data 924 by multiplying the determination result of the "bright" and "dark" areas of the video see-through image data 924 by the identification result of the relatively high brightness areas, such as blown-out highlights. Warp18 sets "0" to areas of the video see-through image data 924 that it determines not to be displayed (for example, areas that fall into a "bright place"). Therefore, in the video see-through image data 924a after brightness adjustment, areas that fall into a "bright place" in any of the three pattern determinations are set to "0".

In addition, the Display Controller 19 begins providing visual assistance using video see-through image data not only in "dark places" but also in "twilight" areas. Unlike "dark places," such "twilight" areas allow the user to faintly see the transmitted image. Therefore, in order to prevent double images, the Warp 18 uses the first positional relationship data to deform the video see-through image data for at least the "twilight" area. This deformation causes the position and size of the subject in the transmitted image to match the subject in the video see-through display image on the transmissive display 3. Note that in "dark places," the transmitted image is not visible and no double images occur, so deformation of the video see-through image data can be omitted. In this case, the amount of calculation required for deformation can be reduced. However, the user may experience discomfort at the boundary between the "dark place" and the area outside the "dark place," as well as at the boundary between the inside and outside of the transmissive display 3. For this reason, it is preferable for Warp 18 to target both "dark" and "twilight" areas of the video see-through image data that are displayed on the transmissive display 3 as a video see-through display image for deformation.

FIG. 26 shows an example of video see-through image data 925, 925a before and after correction according to the first embodiment. As shown in FIG. 26, Warp18 deforms the "twilight" area of the video see-through image data 925. An example is shown in which areas in "dark places" that do not overlap with the transmitted image are not deformed, but it is also possible to deform both "dark places" and "twilight" areas. Also, as shown in FIG. 26, Warp18 sets the value "0" to "bright places" areas that are not subject to correction.

In this way, the SoC 100a of this embodiment extracts and deforms at least a partial area of the video see-through image data based on the first positional relationship data. Furthermore, the SoC 100a of this embodiment generates display data so that the outline of the subject included in the transmitted image and the outline of the subject included in part of the video see-through image data are continuous on the transmissive display 3, and displays the generated display data on the transmissive display 3. Therefore, the SoC 100a of this embodiment can improve visibility at the boundary between the transmitted image and the video see-through display image when displaying a video see-through display image on the transmissive display 3.

In addition, the SoC 100a of this embodiment calculates feature points of the transmitted image based on the video see-through image data, and generates first positional relationship data based on the feature points of the transmitted image and the feature points of the video see-through image data. The SoC 100a outputs the generated first positional relationship data to memory. Therefore, according to the SoC 100a of this embodiment, the first positional relationship data generated and stored in advance is used when the user uses the head-mounted display 1a, thereby improving the processing speed for correction.

The SoC 100a of this embodiment also extracts a first feature point of the transmitted image from first calibration image data 901 captured by the calibration camera 42 from the front surface 301 of the transmissive display 3. The SoC 100a also extracts a second feature point of the subject from second calibration image data 902 captured by the calibration camera 42 of the transmissive display 3, on which display data based on the video see-through image data captured by the video see-through camera 41 is displayed. The SoC 100a of this embodiment then generates first positional relationship data based on the positional relationship between the first and second feature points. Therefore, the SoC 100a of this embodiment can correct the misalignment between the video see-through image data and the transmitted image by taking into account actual lens distortion, display system distortion, etc.

Furthermore, as described above, the head-mounted display 1a of this embodiment has the functions of providing visual assistance in dark places and suppressing reduced visibility due to glare in bright places. Therefore, for example, when a user wears the head-mounted display 1a while driving a vehicle, visual assistance is provided by the display of a video see-through display image in dark places such as behind objects, and visual assistance is provided by the light blocking liquid crystal shutter in bright places such as backlight. Furthermore, the head-mounted display 1a of this embodiment can also be used as a visual assistance when working in dark places such as at night or underground.

(Modification 1 of the First Embodiment)
In the first embodiment described above, the SoC 100a reduced the occurrence of double images, for example, in the "dark" and "twilight" areas on the transmissive display 3 by transforming the video see-through image data to match the transmitted image.

Another way to make double images less noticeable is to increase the degree of light blocking by the LCD shutter to reduce the effects of external light, and then display video see-through image data. With this method, the transmitted image becomes invisible due to the light blocking, thereby preventing double images.

Figure 27 is a diagram showing an example of a light-blocking target area 201 according to variant 1 of the first embodiment. Generally, the transparent display 3 has a narrower angle of view than the lens 2, and therefore can often only display a portion of the user's field of view. For this reason, the Display Controller 19 of this variant does not block the entire field of view of the user (the entire lens 2), but rather sets only the area where the transparent display 3 is present as the light-blocking target area 201. By setting this area as the light-blocking target area 201, the Display Controller 19 can prevent double images without obstructing the field of view outside the range of the transparent display 3.

(Modification 2 of the First Embodiment)
In the first modification of the first embodiment described above, the SoC 100a defines only the area of the entire lens 2 where the transmissive display 3 is present as the light-shielded area 201. In such a light-shielded area, if there is an area where the relative or absolute value of the EV value is high, such as in strong backlight, it is not possible to reduce glare outside the area of the transmissive display 3.

FIG. 28 is a diagram showing an example of a case where a bright area exists according to Variation 2 of the first embodiment. In such a case, even if SoC 100a uses the liquid crystal shutter to block light only in the area where the transmissive display 3 exists, the user will still feel dazzled. Furthermore, in the example shown in FIG. 28, there is also a dark area within the field of view of lens 2. Even in dark areas, areas outside the range of the transmissive display 3 are not subject to visual aid by the video see-through display image. Therefore, when SoC 100a blocks light from the entire surface of lens 2 using the liquid crystal shutter, areas outside the range of the transmissive display 3 remain dark, resulting in poor visibility.

In such cases, the SoC 100a of this modified example controls the liquid crystal shutter to block light only in areas that are dazzling due to strong backlight, etc.

FIG. 29 is a diagram showing an example of a light-shielding target area 202 according to Modification 2 of the first embodiment. As shown in FIG. 29, the light-shielding target area 202 also includes areas outside the range of the transmissive display 3. For example, the CPU 23 of the SoC 100a of this modification calculates the transmittance for each area of the lens 2 from the average brightness value for each rectangular block, and controls the transmittance of the liquid crystal shutter for each area. The liquid crystal shutter of the lens 2 of this modification has a liquid crystal panel whose transmittance can be partially controlled.

By controlling the light-blocking range in this manner, the head-mounted display 1a of this modified example can reduce the reduction in visibility for the user when there is a bright area with relatively or absolutely high brightness outside the range of the transmissive display 3 within the field of view of the lens 2. Furthermore, the head-mounted display 1a of this modified example can reduce the reduction in visibility for the user even when there is a mixture of bright and dark areas within the field of view of the lens 2.

Second Embodiment
In the first embodiment described above, the SoC 100a corrects the misalignment between the video see-through display image and the transmitted image based on the difference between the position of the user's eye 8 wearing the head-mounted display 1a and the mounting position of the video see-through camera 41. In the calibration process of the first embodiment, the SoC 100a determines the first positional relationship data on the assumption that the user's eye 8 is located near the center of the lens 2. However, the position of the eye 8 varies depending on the individual user, and even for the same person, the relative position between the eye 8 and the video see-through camera 41 is expected to constantly change depending on the wearing conditions. For this reason, in this second embodiment, the misalignment during mounting is also corrected.

Figure 30 is a diagram showing an example of the overall configuration of a head-mounted display 1b according to the second embodiment. The head-mounted display 1b of this embodiment includes an eyeglass body 10, lenses 2a and 2b, transparent displays 3a and 3b, video see-through cameras 41a and 41b, display projectors 5a and 5b, an ambient light sensor 60, head tracking cameras 63a and 63b, and an SoC 100b. The head-mounted display 1b of this embodiment also includes eye tracking cameras 43a and 43b.

The eyeglasses main body 10, lenses 2a, 2b, transmissive displays 3a, 3b, video see-through cameras 41a, 41b, display projectors 5a, 5b, ambient light sensor 60, and head tracking cameras 63a, 63b have the same functions as in the first embodiment. Also, the head-mounted display 1b has an IMU 61, a ToF sensor 62, a flash memory 31, and a DRAM 32, just like in the first embodiment.

Eye Tracking cameras 43a and 43b capture images in the direction in which the front of the transmissive display 3 faces.

Figure 31 is a diagram showing an example of the positional relationship between the eye tracking cameras 43a, 43b and the user's eyes 8a, 8b according to the second embodiment. As shown in Figure 31, the eye tracking cameras 43a, 43b capture images of the eyes 8a, 8b of a user wearing the head-mounted display 1b.

Note that the number of eye tracking cameras 43a, 43b is not limited to two. Figure 32 is a diagram showing another example of the positional relationship between the eye tracking cameras 43a-43d and the user's eyes 8a, 8b according to the second embodiment. As shown in Figure 32, when the head-mounted display 1b is equipped with four eye tracking cameras 43a-43d, it is possible to capture images of one eye 8 using two cameras. With this configuration, it is also possible to estimate the distance from the eyes 8a, 8b to the lenses 2a, 2b using the principle of triangulation.

Hereinafter, when there is no need to distinguish between the individual eye tracking cameras 43a to 43d, they will simply be referred to as eye tracking cameras 43. Eye tracking camera 43 is an example of the third camera in this embodiment. Furthermore, image data captured by eye tracking camera 43 is an example of the fourth image data in this embodiment.

FIG. 33 is a diagram showing an example of the configuration of SoC 100b according to the second embodiment. Like the first embodiment, SoC 100b includes I2C interfaces 11 and 22, Mono ISP 12, DSP & AI Accelerator 13, SRAMs 14a to 14c, GPU 15, Color ISP 16, Time Warp 17, Warp 18, Display Controller 19, STAT 21, CPU 23, and DRAM Controller 24.

In addition to the functions of the first embodiment, the Mono ISP 12 of this embodiment also acquires captured image data from the eye tracking camera 43 and corrects brightness, etc.

In addition to the functions of the first embodiment, the DSP & AI Accelerator 13 of this embodiment also performs eye tracking processing to detect the position of the pupil of the user's eye 8 based on image data captured by the eye tracking camera 43 acquired and corrected by the Mono ISP 12. The DSP & AI Accelerator 13 is an example of an pupil detection circuit in this embodiment. Of the DSP & AI Accelerator 13, the DSP 131 in particular generates second positional relationship data based on the position of the user's pupil and feature points of the video see-through image data, and outputs it to a memory such as the SRAM 14a. The DSP 131 is an example of an image processing circuit in this embodiment.

In addition to the functions of the first embodiment, Warp 18 of this embodiment also corrects the position of the subject in the video see-through image data based on second positional relationship data. The second positional relationship data represents the positional relationship between the position of the user's pupil and the video see-through camera 41.

The CPU 23 performs the pre-use calibration process for the head-mounted display 1b in the same manner as in the first embodiment, and stores the first positional relationship data in memory.

Next, the flow of the process for obtaining the pupil position misalignment correction amount in this embodiment will be described. Figure 34 is a flowchart showing an example of the flow of the process for obtaining the pupil position misalignment correction amount according to the second embodiment.

First, Mono ISP 12 acquires image data of the eyes of a user (wearer) wearing head-mounted display 1b from eye tracking camera 43 (S31). The image data captured by eye tracking camera 43 is referred to as eye tracking image data.

Then, the DSP & AI Accelerator 13 detects the position of the user's pupil from the image data for eye tracking (S32). The data indicating the position of the user's pupil is an example of pupil position data indicating the position of the user's pupil.

Then, the DSP in the DSP & AI Accelerator 13 acquires the relative position between the user's eye 8 and the video see-through camera 41 based on the detection result of the user's pupil position (S33). The DSP 131 outputs second positional relationship data indicating the acquired relative position between the user's eye 8 and the video see-through camera 41 to a memory such as the SRAM 14a. The DSP 131 may also generate second positional relationship data indicating the relative position between the user's eye 8 and the video see-through camera 41 based on feature points extracted from the video see-through image data. The DSP 131 may also store pupil position data indicating the position of the user's pupil in a memory such as the SRAM 14a. At this point, the processing of the flowchart in Figure 34 ends.

The method of eye tracking processing is not particularly limited, and any known method can be used. For example, data indicating the positional relationship between the video see-through camera 41 and the eye tracking camera 43 may be stored in advance in a memory such as RAM 14a. In this case, the DSP 131 may convert the relative relationship between the user's eyes 8 and the eye tracking camera 43 into the relative position between the user's eyes 8 and the video see-through camera 41 based on the data indicating this positional relationship. The processing of the flowchart in Figure 34 is repeatedly executed while the head-mounted display 1b is being used by the user.

FIG. 35 is a flowchart showing an example of the flow of the display data generation process when using the head-mounted display 1b according to the second embodiment. The image data acquisition process in S21 and the bright and dark area extraction process in S23 are the same as the processes in the first embodiment described in FIG. 12.

Furthermore, in the process of acquiring positional relationship data of the subject in the image data in S22, Warp 18 acquires first positional relationship data and second positional relationship data from a memory such as SRAM 14a.

Warp 18 then corrects the positional relationship of the subject in the video see-through image data based on the second positional relationship data (S41). The processes from the transformation process of the video see-through image data in S24 to the output process of the display data and transmittance data in S27 are the same as those in the first embodiment described in FIG. 12. At this point, the processing of this flowchart ends.

In this way, the SoC 100b of this embodiment detects the position of the user's pupils based on the Eye Tracking image data, and generates second positional relationship data that represents the positional relationship between the pupil positions and the video see-through camera 41. Therefore, the SoC 100b of this embodiment can reflect the deviation in pupil position when actually worn and individual differences between users in the video see-through display image displayed on the transmissive display 3.

(Modification 1 of the second embodiment)
In the second embodiment described above, the DSP&AI accelerator 13 detects the position of the user's pupil from the eye tracking image data. In this modification, the DSP&AI accelerator 13 extracts pupil reflection image data reflected in the user's pupil from the eye tracking image data. The DSP&AI accelerator 13 is an example of a pupil reflection image extraction circuit in this modification.

In addition, the DSP 131 of this modified example generates second positional relationship data based on the pupil reflection image data and feature points of the video see-through image data, and outputs the data to a memory such as the SRAM 14a. The DSP 131 is an example of an image processing circuit in this modified example.

SoC100b of this modified example extracts pupil reflection image data reflected in the user's eyes from the eye tracking image data captured by the eye tracking camera 43, and generates second positional relationship data based on that pupil reflection image data. Therefore, SoC100b of this modified example can correct video see-through image data by taking into account the image that the user is actually viewing.

(Other Modifications)
In the above-described embodiments, the SoCs 100a and 100b are mounted on the head-mounted displays 1a and 1b, but the SoCs 100a and 100b may be applied to other display devices. For example, the SoCs 100a and 100b may be display devices for head-up displays mounted on the windshield of a vehicle.

Furthermore, in each of the above-described embodiments, the head-mounted displays 1a and 1b may be goggle-type devices equipped with two lenses 2a and 2b, or may be goggle-type devices equipped with one large lens 2 for both eyes. Furthermore, the see-through display 3 may be provided on the entire surface of the lens 2, rather than as part of it.

The various processes executed by SoC 100a, 100b in the above-described embodiments and their variations may be stored, for example, as a program on a non-volatile storage medium. For example, the CPU 23 of SoC 100a, 100b may read the program to execute the various processes described above.

Although the above describes each embodiment and its variations, the image processing device, image processing method, and image processing program disclosed herein are not limited to the above-described embodiments, and the components can be modified and embodied at each implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all of the components shown in the embodiments.

1a, 1b Head-mounted display 2, 2a, 2b Lens 3, 3a, 3b See-through display 5, 5a, 5b Display projector 8, 8a, 8b Eye 9a, 9b Subject 10 Glasses body 11, 22 I2C interface 12 Mono ISP
13 DSP&AI Accelerator
14a-14c SRAM
15 GPUs
16 Color ISP
17 Time Warp
18 Warp
19 Display Controller
21 STAT
23 CPU
24 DRAM Controller
31 Flash memory
32 DRAM
41, 41a, 41b Video see-through cameras 42, 42a, 42b Calibration cameras 43, 43a, 43b Eye tracking cameras 60 Ambient Light Sensor
61 IMU
62 ToF sensor 63, 63a, 63b Head tracking camera 71 Dark place 72 Light place 90 Feature point 100a, 100b SoC
131 DSP
132 AI Acceler
191 EN block 192 Blend block 201, 202 Light-shielded target area 301 Front surface 302 Back surface 310 Area 320 Area 901 First calibration image data 902 Second calibration image data 911 Pseudo-transmitted image data 921 to 925, 921a, 922a, 924a, 925a Video see-through image data 931 to 934, 931a Video see-through display image 940, 940a Pseudo-see-through display image data 941 Transmitted image

Claims (12)

a memory that stores first positional relationship data indicating a positional relationship between a subject included in a transmission image transmitted through a transmissive display and the subject included in first image data captured by a first camera that captures an image in a direction in which the back surface of the transmissive display faces;
a display data generating circuit that extracts and deforms at least a partial area of the first image data based on the first positional relationship data, and generates display data so that the contour of the subject included in the transmission image and the contour of the subject included in the area of the first image data are continuous on the transmissive display;
a display control circuit that displays the display data on the transmissive display;
A semiconductor device having: an image processing circuit that calculates feature points of the transmission image based on the first image data, generates the first positional relationship data based on the feature points of the transmission image and the feature points of the first image data, and outputs the first positional relationship data to the memory;
The semiconductor device according to claim 1 . The image processing circuit
extracting a first feature point of the transmission image from second image data obtained by photographing the subject located on the rear side of the transmission display through the transmission display with a second camera from the front side of the transmission display;
extracting second feature points of the subject from third image data obtained by capturing an image of the transmissive display, on which the display data based on the first image data captured by the first camera is displayed, using the second camera;
generating the first positional relationship data based on a positional relationship between the first feature point and the second feature point;
The semiconductor device according to claim 2 . an image processing circuit that extracts a bright area of the first image data and calculates a bright area of the transmission image based on the bright area of the first image data and the first positional relationship data;
the display data generation circuit generates the display data by deforming a portion of the extracted first image data that is in contact with a bright area of the transmission image.
The semiconductor device according to claim 1 . an image processing circuit that extracts a dark area of the first image data and calculates a dark area of the transmission image based on the dark area of the first image data and the first positional relationship data;
the display control circuit extracts a portion of the first image data corresponding to a dark area of the transmission image and adjusts brightness thereof;
The semiconductor device according to claim 1 . an image processing circuit that extracts a bright area of the first image data and calculates a bright area of the transmission image based on the bright area of the first image data and the first positional relationship data;
the display control circuit controls the transmittance of a region of the transmissive display corresponding to a bright region of the transmission image to a second transmittance that is smaller than a first transmittance used when the image processing circuit calculates the bright region of the transmission image;
The semiconductor device according to claim 1 . a pupil detection circuit that detects the position of a pupil of a viewer of the transmissive display based on fourth image data captured by a third camera that captures an image in a direction in which the front surface of the transmissive display faces;
an image processing circuit that generates second positional relationship data representing a positional relationship between the position of the pupil and the first camera based on the position of the pupil and the feature points of the first image data, and outputs the second positional relationship data to the memory;
The semiconductor device according to claim 1 . a pupil reflection image extraction circuit that extracts pupil reflection image data reflected in the pupil of an observer of the transmissive display from fourth image data captured by a third camera that captures an image in a direction in which the front surface of the transmissive display faces;
an image processing circuit that generates second positional relationship data representing a positional relationship between the position of the pupil and the first camera based on the pupil reflection image data and feature points of the first image data, and outputs the second positional relationship data to the memory;
The semiconductor device according to claim 1 . the memory further stores pupil position data indicating the positions of the pupils of a viewer of the transmissive display;
an image processing circuit that generates second positional relationship data representing a positional relationship between the pupil position and the first camera based on the pupil position data and the feature points of the first image data, and outputs the second positional relationship data to the memory;
The semiconductor device according to claim 1 . the display data generation circuit further corrects the position of the subject in the first image data based on the second positional relationship data.
The semiconductor device according to claim 7 . a first positional relationship data storage step for storing first positional relationship data indicating a positional relationship between a subject included in a transmission image transmitted through a transmissive display and the subject included in first image data captured by a first camera that captures an image in a direction in which the back surface of the transmissive display faces;
a display data generating step of extracting and deforming at least a partial area of the first image data based on the first positional relationship data, and generating display data so that an outline of the subject included in the transmission image and an outline of the subject included in the area of the first image data are continuous on the transmission display;
a display control step of displaying the display data on the transmissive display;
A method comprising: at least one see-through display;
a display projector that displays display data on the transmissive display;
a memory that stores first positional relationship data indicating a positional relationship between a subject included in a transmission image transmitted through the transmissive display and the subject included in first image data captured by a first camera that captures an image in a direction in which the back surface of the transmissive display faces;
a display data generating circuit that extracts and deforms at least a partial area of the first image data based on the first positional relationship data, and generates the display data so that the contour of the subject included in the transmission image and the contour of the subject included in the area of the first image data are continuous on the transmissive display;
a display control circuit that displays the display data on the transmissive display;
A head-mounted display comprising: