WO2024202682A1 - Display device and display method - Google Patents

Abstract

The purpose of the present invention is to provide a display device capable of accurately detecting the positions of a light projection system and an eyepiece optical system relative to each other.　The display device according to the present invention comprises a light projection system that includes a light source, and an eyepiece optical system that guides light projected from the light projection system to an eyeball, the eyepiece optical system including an optical element and three or more feature points, the positions of which relative to the optical element are fixed, and the light projection system including an acquisition unit that acquires at least two-dimensional position information of each of at least three feature points among the three or more feature points. Through the display device according to the present invention, it is possible to provide a display device capable of accurately detecting the positions of the light projection system and the eyepiece optical system relative to each other.

Description

Display device and display method

The technology disclosed herein (hereinafter also referred to as "the technology") relates to a display device and a display method.

　Display devices that display images by irradiating light (more specifically, image light) onto the observer's eyeball are known (see, for example, Patent Document 1).

For example, the display device described in Patent Document 1 detects the relative positions of the scanning unit (light projection system) and the deflection unit (ocular optical system) from two-dimensional position information of each of two feature points on the contact lens, and controls the scanning unit based on the detection results.

International Publication No. 2009/066446

However, conventional display devices have room for improvement in terms of accurately detecting the relative position between the light projection system and the eyepiece optical system.

The main objective of this technology is to provide a display device that can accurately detect the relative position between the light projection system and the eyepiece optical system.

The present technology includes a light projection system including a light source;
an eyepiece optical system that guides the light projected from the light projection system to an eyeball;
Equipped with
The eyepiece optical system includes:
An optical element;
Three or more feature points having fixed relative positions with respect to the optical element;
Including,
The display device further includes an acquisition unit that acquires at least two-dimensional position information of at least three of the three or more feature points.
The at least three feature points may not be collinear.
The optical element may be provided on a contact lens that is fitted to the eye. The at least three features may be provided on the contact lens.
The acquisition unit may be capable of acquiring at least the two-dimensional position information of each of the at least three feature points even if a direction of the eyeball changes.
The optical element may be provided on a lens attached to an eyeglass frame.
The at least three feature points may all be provided on the lens or the eyeglass frame, or some of the at least three feature points may be provided on the lens and others may be provided on the eyeglass frame.
The at least three feature points and the optical center of the optical element may be on the same plane.
The acquisition unit may include at least one camera.
The at least one camera may include a visible light camera and/or a Time-of-Flight camera.
The at least one camera may include a plurality of cameras constituting a stereo camera.
The optical projection system may include the acquisition unit.
The capture unit may include an infrared light source, the camera may be sensitive to infrared wavelengths, and the at least three features may be made of a retroreflective material.
The optical projection system may include a projection optical system disposed on an optical path of light from the light source, and an optical axis of the projection optical system may be coaxial with an optical axis of the camera.
The optical projection system may include a projection optical system arranged on an optical path of light from the light source, and a control unit that controls the light source and/or the projection optical system based on the acquisition result of the acquisition unit.
The projection optical system includes a movable deflection element and a focus lens, and the acquisition unit acquires two-dimensional position information of each of the at least three feature points in a plane perpendicular to the optical axis direction of the camera, and distance information between the camera and each of the at least three feature points in the optical axis direction of the camera, and the control unit may simultaneously control the movable deflection element based on the two-dimensional position information, control the light source based on at least the two-dimensional position information and the distance information, and/or control the focus lens based on the distance information.
At least two of the at least three characteristic points may be integrally provided.
The optical projection system and the eyepiece optical system may be separate entities.
The present technology includes a step of acquiring at least two-dimensional position information of at least three feature points among three or more feature points whose relative positions with respect to an optical element that guides light projected from a light projection system to an eyeball are fixed;
and controlling a part of the optical projection system based on the results obtained in the obtaining step.
The optical projection system may include a light source, a movable deflection element, and a focus lens, and the step of acquiring the at least two-dimensional position information may include a step of acquiring two-dimensional position information of each of the at least three feature points in a plane perpendicular to the optical axis direction of the camera, and a step of acquiring distance information between the camera and each of the at least three feature points in the optical axis direction of the camera, and the step of controlling may include a step of controlling the movable deflection element based on at least the two-dimensional position information, and a step of controlling the light source based on the two-dimensional position information and the distance information and/or a step of controlling the focus lens based on the distance information.

FIG. 1 is a diagram for explaining the principle of the present technology. FIG. 11 is a diagram for explaining a method of calculating a normal vector. 1 is a diagram showing a display device according to a first embodiment of the present technology; 4A to 4C are diagrams for explaining a first configuration example of the eyepiece optical system of the display device of FIG. 5A to 5C are diagrams for explaining a second configuration example of the eyepiece optical system of the display device of FIG. 6A to 6C are diagrams for explaining a third example configuration of the eyepiece optical system of the display device of FIG. 7A to 7E are diagrams for explaining a fourth example configuration of the eyepiece optical system of the display device of FIG. 8A to 8E are diagrams for explaining a fifth example configuration of the eyepiece optical system of the display device of FIG. 1A and 1B are diagrams for explaining the incidence angle dependency of the diffraction efficiency in a HOE. FIG. 4 is a block diagram showing an example of a basic configuration for controlling the display device shown in FIG. 3 . 11 is a flowchart showing the flow of overall control in the basic configuration example of FIG. 10 . FIG. 4 is a block diagram showing a first example of a control configuration for the display device shown in FIG. 3 . 13 is a flowchart showing the flow of overall control in the configuration example 1 of FIG. 12 . 13 is a flowchart showing a movable mirror control process in the configuration example 1 of FIG. 12 . 13 is a flowchart showing a focus lens control process in the configuration example 1 of FIG. 12 . 13 is a flowchart showing a projection image control process in the configuration example 1 of FIG. 12 . FIG. 4 is a block diagram showing a second example of a control configuration for the display device shown in FIG. 3 . 18 is a flowchart showing the flow of overall control in the configuration example 2 of FIG. 17 . FIG. 11 is a diagram showing a display device according to a second embodiment of the present technology. 10 is a flowchart showing a flow of control of a display device according to a second embodiment of the present technology. FIG. 13 is a diagram showing a display device according to a third embodiment of the present technology. FIG. 13 is a diagram showing a display device according to a fourth embodiment of the present technology. FIG. 13 is a diagram showing a display device according to a fifth embodiment of the present technology. FIG. 1 is a diagram for explaining the principle of a stereo camera. FIG. 13 is a diagram showing a display device according to a sixth embodiment of the present technology. FIG. 1 is a diagram for explaining retroreflection. FIG. 2 is a block diagram showing an example of the configuration of a TOF camera. FIG. 2 is a block diagram showing an example of the configuration of an event camera. 29A and 29B are diagrams for explaining configuration examples 6 and 7 of the eyepiece optical system of the display device in FIG.

Below, a preferred embodiment of the present technology will be described in detail with reference to the attached drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be denoted with the same reference numerals to avoid repeated description. The embodiment described below shows a representative embodiment of the present technology, and is not intended to narrow the scope of the present technology. Even if this specification describes that the display device and display method related to the present technology have multiple effects, it is sufficient that the display device and display method related to the present technology have at least one effect. The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

The explanation will be given in the following order:
0. Introduction 1. Display device according to a first embodiment of the present technology 2. Display device according to a second embodiment of the present technology 3. Display device according to a third embodiment of the present technology 4. Display device according to a fourth embodiment of the present technology 5. Display device according to a fifth embodiment of the present technology 6. Display device according to a sixth embodiment of the present technology 7. Effects of the display device according to the present technology 8. Modifications of the present technology

<0. Introduction>
Conventionally, there is known a retinal direct imaging device (display device) that projects light (specifically, image light) from an optical projection system and directs the projected light to the pupil of an observer via an eyepiece optical system worn by the observer to display an image. In particular, to realize a retinal direct imaging device in which the positional relationship (relative position) between the optical projection system and the eyepiece optical system is not fixed, it is necessary to recognize the state of the observer from the optical projection system side. Such a retinal direct imaging device has a feature that the relative position between the eyepiece optical system and the optical projection system changes in real time depending on the movement of the observer, so it is mainly desired to perform the following recognition and correction.
- Recognizing the position of the optical center of the optical element as seen from the light projection system In the case of Maxwell optical systems, the observer cannot see the image unless the projection optical axis passes through the pupil. Normally, optical elements are positioned so that their optical centers overlap with the observer's pupil, so it is desirable to recognize the position of the optical center of the optical element (e.g., two-dimensional position information) from the light projection system.
Correction of the angle (attitude) of the optical element relative to the projection optical axis The diffraction efficiency of a diffraction element (e.g., HOE: hologram element) as an optical element depends on the angle of incidence, and when the angle of incidence of the projection light to the HOE changes, the diffraction efficiency changes independently for each of RGB. In order to correctly reproduce the image and brightness that you want the viewer to see, it is desirable to correct the brightness of the original image depending on the angle of incidence.

Incidentally, in the display device described in Patent Document 1, the relative position between the scanning unit (light projection system) and the deflection unit (eyepiece optical system) is detected from two-dimensional position information of each of two feature points provided in the deflection unit (eyepiece optical system). However, there is room for improvement in the detection accuracy of this relative position in this display device.

After extensive research, the inventors developed a display device according to this technology that can accurately detect the relative position between the light projection system and the eyepiece optical system (for example, the relative position of the eyepiece optical system as viewed from the light projection system).

In particular, in a display device according to this technology, when the optical elements of the eyepiece optical system are mounted on a contact lens that is worn on the observer's eyeball, the contact lens and the optical elements move in accordance with the movement of the pupil, so that if the position of the pupil is recognized, the position of the optical element can also be recognized, and if the direction of the pupil (direction of gaze) is recognized, the angle (posture) of the optical element can also be recognized. This makes it possible to detect the relative position between the light projection system and the eyepiece optical system with greater precision.

In addition, the display device described in Patent Document 1 is thought to be configured so that the scanning unit (light projection system) is placed on the observer's head, and it would be extremely difficult to accommodate a configuration requiring high detection accuracy in which the scanning unit (light projection system) and the deflection unit (ocular optical system) are arranged separately and spatially separated by a certain long distance, causing their positional relationship to change dynamically.

On the other hand, the display device according to this technology can accurately detect changes in the relative position between a spatially separated (separate) optical projection system and an eyepiece optical system, and is therefore fully capable of handling configurations in which the positional relationship between such an optical projection system and an eyepiece optical system changes dynamically.

1 is a diagram for explaining the principle of this technology. As shown in FIG. 1, in this technology, the eyepiece optical system EOS has three or more (for example, three) feature points FP whose relative positions with respect to the optical element OE are fixed. In the example of FIG. 1, three feature points FP are provided on the spectacle lens GL on which the optical element OE is provided. At least one of the three feature points FP may be provided, for example, on the spectacle frame GF. These three feature points FP are recognized by a camera C mounted on a light projection system LPS that is spatially separated from the eyepiece optical system EOS. By measuring the spatial coordinates and calculating the normal vector as follows, the three-dimensional coordinates (x, y, z) of the optical center OC (optical center) of the optical element OE as seen from the light projection system LPS and the angle (θx, θy) of the optical element OE as seen from the light projection system LPS can be detected in real time. Note that θx indicates the rotation angle (pitch) around the x-axis extending to the left and right of the observer. θy indicates the rotation angle (yaw) around the y-axis that extends above and below the observer. In this specification and drawings, unless otherwise specified, "x" and "X" are used with the same meaning, "y" and "Y" are used with the same meaning, and "z" and "Z" are used with the same meaning.

(Measurement of spatial coordinates)
For example, the three-dimensional coordinates (x, y, z) of each of the three feature points FP are obtained. If the positional relationship between the three feature points FP and the optical center OC of the optical element OE is known in advance, the three-dimensional coordinates (x, y, z) that are the relative position of the optical center OC of the optical element OE as seen from the light projection system LPS can be calculated by interpolation or extrapolation of the three points. The three-dimensional coordinates (x, y, z) of each feature point FP are obtained using a camera C. The main acquisition method is the stereo camera method, but a configuration using only a TOF camera or a configuration using a visible light camera and a TOF camera in combination is also possible. In addition to a visible light camera, the stereo camera method also includes a method using an infrared camera, an infrared light source, and a retroreflective material.

(Calculation of normal vector)
2 is a diagram for explaining a method of calculating a normal vector. From the three-dimensional coordinates (x, y, z) of each of the three feature points FP obtained as described above, a normal vector of a plane including these three feature points FP is calculated using a vector cross product calculation. This normal vector corresponds to the orientation of the optical element OE, and the angle (θx, θy) of the optical element OE as viewed from the optical projection system LPS can be calculated by taking the inner product with the axis vector of the reference coordinate system set in the optical projection system LPS.

If the spatial coordinate vectors of the three feature points FP are P[0], P[1], and P[2], respectively, the normal vector N of the plane containing these three feature points can be calculated using the cross product of vectors as shown in the following equation (1).
N = (P[2]-P[0]) × (P[1]-P[0]) ... (1)

The angles θx, θy, and θz between the x-axis, y-axis, and z-axis and the normal vector N can be calculated using the dot product of vectors as shown in the following equations (2) to (4), assuming that the directional vectors of each axis are Ex, Ey, and Ez.
N・Ex = |N||Ex|cos θx...(2)
N・Ey = |N||Ey|cos θy...(3)
N・Ez = |N||Ez|cos θz...(4)

From the above equations (2) to (4), θx, θy, and θz can be calculated using the following equations (5) to (7).
θx = arccos (cos θx) ...(5)
θy = arccos (cos θy) ...(6)
θz = arccos (cos θz) ...(7)

To perform the above calculations, the three-dimensional coordinates of at least three feature points FP are required. If there are four or more feature points FP, at least three of them can be selected to perform the calculations.

<1. Display device according to first embodiment of the present technology>
FIG. 3 is a diagram showing a display device 1 according to a first embodiment of the present technology. As an example, the display device 1 is a retinal direct imaging type display device that directly images an image on the retina of a user who is an observer by light. The display device 1 includes a light projection system 10 and an eyepiece optical system 20 that guides light projected from the light projection system 10 to an eyeball EB. The light projection system 10 further includes an acquisition unit 300 including at least one camera 300a and a control unit 400 (see 10, for example). Note that the acquisition unit 300 does not have to be included in the light irradiation system 10. Specifically, the acquisition unit 300 may be provided separately from the light irradiation system 10 (spatially separated). Even in this case, it is desirable that the positional relationship between the light irradiation system 10 and the acquisition unit 300 remains unchanged.

In the display device 1, as an example, the optical projection system 10 and the eyepiece optical system 20 are provided separately (spatially separated). The optical projection system 10 may be placed, for example, on a desk, a table, etc., or hung on a wall, or may be attached to a portable item such as a user's wristwatch or smartphone, or may be attached to a moving object ridden by the user, such as a car, motorcycle, or bicycle. The eyepiece optical system 20 is provided at or near the observer's eyeball EB.

Image data is input to the control unit 400. The control unit 400 generates modulation signals for each color of RGB according to the input image data, and outputs the modulation signals to a light source driving unit 100d (described later) of the light projection system 10 (see, for example, FIG. 12).

In FIG. 3, the optical projection system 10 has two cameras 300a, but it may have only one camera 300a, or it may have three or more cameras 300a. (The same applies to other figures in which multiple cameras 300a are shown.)

(Light projection system)
The optical projection system 10 includes, for example, an image light generation unit 100 and a projection optical system 200 that projects the image light IL generated by the image light generation unit 100. The image light generation unit 100, the projection optical system 200, and at least one camera 300a are, for example, integrally provided in a housing H.

The image light generating unit 100 includes, as an example, a light source 100a, a scanning mirror 100b, and a light source driving unit 100d (see FIG. 12).

The light source 100a has a laser, such as an edge-emitting laser (LD) or a surface-emitting laser (VCSEL). Here, the light source 100a has, for example, a red laser, a green laser, and a blue laser. Note that, for example, a light-emitting diode (LED) may be used as the light source 100a. The light source 100a may have only a laser or LED of a single color. This allows the configuration of the light source 100a to be simplified.

The light source driving unit 100d generates a driving signal based on the modulation signal for each color of RGB input from the control unit 400, and applies the driving signal to the corresponding laser of the light source 100a to drive the laser. The light source 100a and the light source driving unit 100d are collectively referred to as the "light source system." As an example, the light source driving unit 100d receives luminance information of the projected image from the control unit 400.

Scanning mirror 100b deflects and scans the light emitted from light source 100a. For example, a MEMS mirror that can be driven around two axes is used as scanning mirror 100b. This allows for a reduction in the number of parts and a smaller size. On the other hand, if a mirror that can be driven around two axes is used as scanning mirror 100b, distortion may occur in the projected image due to the driving of the mirror. To avoid this, for example, two MEMS mirrors that can be driven around one axis may be used in combination.

The projection optical system 200 has, as an example, a focus lens 200c as a projection lens and a movable mirror 200a as a movable deflection element. The projection optical system 200 is controlled by the control unit 400. A projection window 200b is provided behind the movable mirror 200a.

The focus lens 200c is disposed on the optical path of the image light IL via the scanning mirror 100b. The focus lens 200c is provided so as to be movable in the optical axis direction, and is driven in the optical axis direction by a focus lens drive unit. The focus lens drive unit is controlled by the control unit 400. If the observer moves in the z-axis direction (a direction parallel to the projection optical axis), the image will not be in focus. Therefore, it is preferable to adjust the position of the focus lens 200c in the optical axis direction (focus adjustment) so that the focus of the image light IL coincides with the observer's retina according to the position of the optical element OE in the z-axis direction.

The movable mirror 200a reflects the image light IL through the focus lens 200c and guides it to the projection window 200b. The image light IL reflected by the movable mirror 200a and passing through the projection window 200b is projected to the eyepiece optical system 20. The movable mirror 200a is arranged to be swingable around two axes, for example, and is driven by a movable mirror drive unit. As the movable mirror 200a, for example, a combination of two galvanometer mirrors that can swing around one axis, or a gimbal mirror that can swing around two axes, etc. are used. The movable mirror drive unit is controlled by the control unit 400. By swinging the movable mirror 200a around two axes, the projection direction of the image light IL can be changed two-dimensionally. The light projection system 10 and the eyepiece optical system 20 are spatially separated, and the relative positional relationship changes dynamically in real time depending on the orientation of the observer's eyeball, head, and body. If the projection optical axis of the light projection system 10 (more specifically, the optical axis of the projection optical system 200, hereinafter also simply referred to as the "projection optical axis") does not pass through the optical element OE, the observer will not be able to see the image. Therefore, it is preferable to control the angle (posture) of the movable mirror 200a so that the projection optical axis passes through the optical element OE based on the two-dimensional coordinates (x, y) of the optical element OE, i.e., based on two-dimensional position information in a plane perpendicular to the projection optical axis. This allows the observer to continue viewing the image at all times. Note that instead of a movable mirror, a mechanical-less deflection element can also be used as the movable deflection element.

(Ocular optical system)
The eyepiece optical system 20 includes an optical element OE and three or more feature points FP. The eyepiece optical system 20 is located within the imaging field of view (within the angle of view) of the camera 300a when the display device 1 is in use (when the image light IL is projected from the optical projection system 10 toward the eyeball EB).

The optical element OE is positioned so that its focal point is located near the observer's pupil. The image light IL projected from the light projection system 10 is focused by the optical element OE and passes through the observer's pupil, thereby being imaged directly on the retina. This allows the observer to recognize the image.

The optical element OE is provided in a contact lens CL that is attached to the observer's eyeball EB. The contact lens CL may or may not be a component of the eyepiece optical system 20. The contact lens CL has the advantage of self-alignment, in that the center of the contact lens CL moves to coincide with the center of the pupil of the eyeball EB. In other words, the orientation of the contact lens CL coincides with the direction of the line of sight, which is the direction of the eyeball EB. For example, when the optical element OE is provided in glasses, there is a possibility that a positional deviation occurs between the glasses and the pupil. In this case, it is necessary to detect the positions of the glasses and the pupil and perform control that takes into account the positional deviation, or to adjust the glasses and drive the changes over time to a level where the amount of positional deviation is not a problem. By providing the optical element OE in the contact lens CL, there is no need to consider this positional deviation. The degree of the contact lens CL may be 0.

The optical element OE is, for example, a holographic element (HOE), which diffracts and focuses the image light IL projected from the optical projection system 10 and guides it to the retina of the eyeball EB. If the angle (posture) of the HOE with respect to the projection optical axis changes, the color of the image seen by the observer changes. This is because the diffraction efficiency of the HOE depends on the angle of incidence.

That is, the HOE has a characteristic that the diffraction efficiency varies depending on the angle of incidence of light. Specifically, the diffraction efficiency of the HOE has an incidence angle dependency such that light other than the image light IL projected from the light projection system 10 is transmitted without being diffracted (see FIG. 9). This allows the observer to see the background image and the projected image superimposed, as in AR. On the other hand, it is physically impossible for the diffraction efficiency of the HOE to be 1 (100%) only in a certain incidence angle range (see the graph on the left side of FIG. 9), and the diffraction efficiency has a characteristic that it always has a peak incidence angle and then drops with an angle width (see the right diagram of FIG. 9). Furthermore, this characteristic is independent of RGB, and the peak angle and half-width do not necessarily match. In order to optimize the color of the image to be shown to the observer through a HOE with such characteristics, it is necessary to recognize the angle (θx, θy) of the HOE with respect to the projection optical axis and generate the projected image taking into account the diffraction efficiency at that angle.

(Feature points)

It is desirable that each feature point FP has different light reflection characteristics from the surrounding areas so that it can be distinguished from the surrounding areas in the image captured by camera 300a. For example, each feature point FP may be made of a reflective material with a higher reflectance than the surrounding areas, or may be made of a light-transmitting or light-absorbing material with a lower reflectance than the surrounding areas. For example, each feature point FP may be made to include irregularities obtained by processing a part of a member (e.g., a contact lens, an eyeglass lens, an eyeglass frame, etc.).

For example, a visible light camera may be used as the camera 300a, and markers visible at visible light wavelengths may be used as each feature point FP. By using a visible light camera, the markers can be recognized over a wide dynamic range. Also, for example, an infrared camera may be used as the camera 300a, and markers visible at infrared wavelengths (e.g., a dichroic film) may be used as each feature point FP.

The relative position of each feature point FP with respect to the optical element OE must be fixed. Therefore, in the display device 1, at least three feature points FP are provided on the contact lens CL, which serves as a rigid body on which the optical element OE is provided. In other words, each feature point FP and the optical element OE are integrated via the contact lens CL.

If the XYZ coordinates of the three feature points FP are P[0] to P[2] (see FIG. 2), the coordinate of the optical center OC of the optical element OE is Hc, and the position vectors of the three feature points FP and the optical center OC are P[0], P[1], P[2], Hc, then the following equation (8) is established using a linear combination of vectors.
Hc = a P[0] + b P[1] + c P[2] ... (8)

Here, if the coordinates Po[0] to Po[2] of each feature point FP in the initial state (at the time of manufacture) and the coordinate Hco of the optical center OC of the optical element OE are known, the coefficients a to c can be found. Furthermore, if each feature point FP can be positioned so that a+b+c=1 holds, then the above formula (8) will always hold no matter what position or attitude the rigid body (e.g., contact lens CL) is in. In other words, the optical center Hc of the optical element OE can be calculated from the three-dimensional coordinates P[0] to P[2] of the three feature points FP recognized by the camera 300a. The condition a+b+c=1 listed here means that the optical center Hc of the optical element exists within a plane specified by the three feature points FP with three-dimensional coordinates P[0] to P[2].

In order for the camera 300a to recognize positional information (x, y, z, θx, θy) about the five axes of the optical element OE, it is desirable that at least three feature points FP are always located in positions where they can be imaged by the camera 300a (however, this excludes times when the observer has their eyes closed or when they blink, as these do not need to be recognized). In other words, it is desirable that the acquisition unit 300, including the camera 300a, be able to acquire at least two-dimensional coordinates (x, y) of each of the at least three feature points FP even if the orientation of the eyeball EB changes. Ideally, it is desirable that the acquisition unit 300, including the camera 300a, be able to acquire at least two-dimensional coordinates (x, y) of each of the at least three feature points FP, regardless of the orientation of the eyeball EB.

Therefore, it is possible to devise the arrangement, size, shape, color, etc. of the three characteristic points FP. In addition, for example, four or more characteristic points FP may be arranged so that even if some of the characteristic points FP are hidden by the eyelids, etc., at least three characteristic points FP can be captured by the camera 300a. In addition, the three characteristic points FP do not necessarily need to be physically separated from each other (separate), and may be provided, for example, integrally with some figure (see Figures 29A and 29B). In the configuration example 6 of the eyepiece optical system 20 shown in Figure 29A, the three characteristic points FP are provided integrally with the circle and spaced apart from each other in the circumferential direction. In the configuration example 7 of the eyepiece optical system 20 shown in Figure 29B, two of the three characteristic points FP are provided integrally with the arc and spaced apart from each other in the circumferential direction, and one characteristic point FP is provided separately from the two characteristic points FP and the arc.

In order to determine the orientation of the optical element OE using the position information of each of the three feature points FP, a plane needs to be specified by at least three feature points FP, so it is necessary that at least three of the three or more feature points FP are not on the same line. In addition, it is preferable that at least three feature points FP and the optical center OC of the optical element OE are on the same plane.

(Example 1 of the eyepiece optical system configuration)
4A to 4C are diagrams for explaining a first configuration example of the eyepiece optical system 20 of the display device 1. In the first configuration example of the eyepiece optical system 20, as shown in Fig. 4A, three feature points FP of the same shape (e.g., circles) are provided on the contact lens CL so as to be located at the three vertices of a triangle and to be located around the pupil (the darkest part in Figs. 4A to 4C). This allows the three feature points FP to be captured by the camera 300a even if the eyeball EB moves left and right to some extent as shown in Figs. 4B and 4C.

(Example 2 of the eyepiece optical system configuration)
5A to 5C are diagrams for explaining a second configuration example of the eyepiece optical system 20 of the display device 1. In the second configuration example of the eyepiece optical system 20, as shown in Fig. 5A, three feature points FP of the same shape (e.g., circles) are located at the three vertices of a triangle, and are provided on the contact lens CL so as to be located around the pupil (the part with the darkest color in Figs. 5A to 5C) and on the inside compared to the first configuration example of Fig. 4A. This allows the three feature points FP to be captured by the camera 300a even if the eyeball EB moves relatively widely left and right as shown in Figs. 5B and 5C.

(Example 3 of the eyepiece optical system configuration)
6A to 6C are diagrams for explaining a configuration example 3 of the eyepiece optical system 20 of the display device 1. In the configuration example 3 of the eyepiece optical system 20, three relatively large feature points FP of different shapes (e.g., circular, triangular, and rectangular) are provided on the contact lens CL so as to be located at the three vertices of a triangle and to be located around the pupil (the part with the darkest color in Figs. 6A to 6C). This allows the three feature points FP to be captured by the camera 300a even if the eyeball EB moves relatively widely left and right as shown in Figs. 6B and 6C. In this configuration example 3, at least two of the feature points FP may be made to have different colors instead of or in addition to the shapes.

(Example 4 of the eyepiece optical system configuration)
7A to 7E are diagrams for explaining a fourth configuration example of the eyepiece optical system 20 of the display device 1. In the fourth configuration example of the eyepiece optical system 20, as shown in Fig. 7A, six feature points FP of the same shape (e.g., circles) are provided on the contact lens CL so as to be located at the six vertices of a hexagon and to be located around the pupil (the part with the darkest color in Figs. 7A to 7E). This allows at least three of the six feature points FP to be captured by the camera 300a even if the eyeball EB moves relatively widely up, down, left, and right as shown in Figs. 7B to 7E.

(Configuration Example 5 of Eyepiece Optical System)
8A to 8E are diagrams for explaining a configuration example 5 of the eyepiece optical system 20 of the display device 1. In the configuration example 5 of the eyepiece optical system 20, as shown in Fig. 8A, three feature points FP of the same shape (e.g., circles) that transmit visible light and reflect infrared light are provided on the contact lens CL so as to be located at the three vertices of a triangle and overlap with the pupil (the darkest part in Figs. 8A to 8E). As a result, even if the eyeball EB moves significantly up, down, left, and right as shown in Figs. 8B to 8E, the three feature points FP can be captured by the camera 300a (e.g., an infrared camera).

(Basic configuration example of control of display device)
Fig. 10 is a block diagram showing a basic configuration example of the control of the display device 1. The acquisition unit 300 of the display device 1 acquires at least two-dimensional position information (e.g., three-dimensional position information) of at least three feature points FP among three or more feature points FP whose relative positions with respect to the optical element OE are fixed. As shown in Fig. 10, the acquisition unit 300 has, as an example, a camera 300a and an xyz coordinate acquisition unit 300b. The control unit 400 has a calculation unit 400a and a display control unit 400b. The acquisition unit 300 and the control unit 400 are realized by hardware such as a CPU, a chipset, etc.

The xyz coordinate acquisition unit 300b acquires the xyz coordinates (three-dimensional position information) of at least three feature points FP from the image captured by the camera 300a.

The calculation unit 400a calculates the angle of the movable mirror 200a, the position of the focus lens 200c, and the brightness of the projected image based on the xyz coordinates of each of at least three feature points FP.

The display control unit 400b controls the movable mirror 200a based on the calculation result of the angle of the movable mirror 200a, controls the focus lens 200c based on the calculation result of the position of the focus lens 200c, and controls the projected image based on the calculation result of the luminance of the projected image (specifically, it outputs a luminance correction value based on the calculation result of the luminance of the projected image to the light source driving unit 100d).

(Overall control flow in basic configuration example of control of display device)
Fig. 11 is a flowchart showing the flow of overall control in the basic configuration example of the control of the display device 1 shown in Fig. 10. Fig. 11 is based on a processing algorithm executed by the CPU. The series of processes in Fig. 11 are started when the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates).

In the first step S1, the acquisition unit 300 acquires the three-dimensional coordinates (x, y, z) of each of at least three feature points FP.

In the next step S2, the calculation unit 400a of the control unit 400 calculates the normal vector N based on the three-dimensional coordinates (x, y, z) of each of the at least three feature points FP.

In the next step S3, the calculation unit 400a of the control unit 400 calculates θx, θy, and θz using the normal vector N.

In the next step S4, the calculation unit 400a of the control unit 400 calculates the distance D from the camera 300a to the optical element OE (more specifically, the plane identified by the three feature points FP) based on the three-dimensional coordinates (x, y, z) of each of the at least three feature points FP.

In the next step S5, the display control unit 400b of the control unit 400 controls the display of the image based on x, y, θx, θy, θz, and D (specifically, controls the movable mirror 200a, the focus lens 200c, and the brightness and shape correction of the projected image).

In the next step S6, the control unit 400 determines whether or not to end the process. For example, when the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of the camera 300a), the control unit 400 denies this determination (determines not to end the process) and performs the series of processes of steps S1 to S5 again. For example, when the user issues an end command or the acquisition unit 300 can no longer acquire the three-dimensional coordinates of each feature point FP (when each feature point FP falls outside the angle of view of the camera 300a), the control unit 400 affirms this determination (determines to end the process) and ends the flow.

(Stereo camera)
A stereo camera method can be introduced, in which multiple cameras 300a are used to obtain the z coordinates of each feature point FP (the distance from the camera 300a to each feature point FP). In the stereo camera method, two cameras 300a are arranged so that their optical axes are parallel to each other, and each parameter is defined as follows (see FIG. 24).

h: Distance between the centers of the left and right cameras [mm]
f: Camera lens focal length [mm]
size_x: Camera sensor size width [mm]
size_y : Camera sensor size (vertical) [mm]
width: Number of pixels across the camera sensor
height: number of vertical pixels of the camera sensor
P: Measurement object coordinates (x, y, z) [mm]
Image point on the right camera sensor of pFP (hr,vr) [pixel]
pl: Image point on the left camera sensor of P (hl,vl) [pixel]

To convert the coordinates of the image point into [mm], xr, yr, and xl are as follows:
xr = hr / ( width × size_x)
yr = vr / ( height × size_y)
xl = hl / ( width × size_x)
xl = vl / ( height × size_y)

From the above, the coordinates of the measurement object P(x, y, z) can be calculated as follows:
x = (xl + xr) * h / 2 * (xl - xr )
y = yr * h / (xl - xr ) = yl * h / (xl - xr )
z = f * h / (xl - xr)

In the above formulas and in Figure 24, the midpoint between the left and right cameras is defined as the origin O. Note that in the stereo camera method, the optical axes of the two cameras do not necessarily need to be parallel, and formulas for non-parallel arrangements are also known, but they will not be discussed here.

(Configuration Example 1 of Display Device Control)
FIG. 12 is a block diagram showing a first example of a control configuration of the display device 1. As shown in FIG.

Configuration example 1 of the control of the display device 1 is a more specific configuration of the control unit 400 compared to the basic configuration example shown in FIG. 10. In this configuration example 1, a stereo camera SC is configured by two cameras 300a (e.g., visible light cameras).

In this configuration example 1, the calculation unit 400a of the control unit 400 has an optical center xyz coordinate calculation unit 400a1 and an optical element tilt angle calculation unit 400a2, and the display control unit 400b of the control unit 400 has a movable mirror control unit 400b1, a focus lens control unit 400b2, and a projection image control unit 400b3.

(Overall control flow in configuration example 1 of the control of the display device)
Fig. 13 is a flowchart showing the flow of the overall control in the configuration example 1 in Fig. 12. Fig. 13 is based on a processing algorithm executed by a CPU. The series of processes in Fig. 13 are started when the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates).

First, in step S21, the acquisition unit 300 performs an xyz coordinate acquisition process. Specifically, the xyz coordinate acquisition unit 300b acquires the xy coordinates and z coordinates of at least three feature points FP from the images captured by the two cameras 300a, 300a that make up the stereo camera SC.

Then, the control unit 400 performs the movable mirror control process in step S22-1, the focus lens control process in step S22-2, and the projection image control process in step S22-3 in parallel.

Next, in step S23, the control unit 400 judges whether or not to end the process. If the judgment here is positive, the flow ends, and if it is negative, the flow returns to step S21 and the same series of processes are performed again. If the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (if each feature point FP is within the angle of view of the camera 300a), the control unit 400 denies the judgment here (determines not to end the process) and performs the processes of steps S21, S22-1, S22-2, and S22-3 again. For example, when the user issues an end command or when the acquisition unit 300 can no longer acquire the three-dimensional coordinates of each feature point FP (if each feature point FP falls outside the angle of view of the camera 300a), the control unit 400 makes a positive judgment here (determines to end the process) and ends the flow.

(Movable mirror control process)
FIG. 14 is a flowchart showing the movable mirror control process (step S22-1) of FIG.

In the first step S22-1-1, the optical center xyz coordinate calculation unit 400a1 calculates the xyz coordinates of the optical center OC of the optical element OE.

In the next step S22-1-2, the movable mirror control unit 400b1 determines the angle of the movable mirror 200a based on the x and y coordinates of the optical center OC. Note that the z coordinate of the optical center OC may also be used in this determination.

In the final step S22-1-3, the movable mirror control unit 400b1 controls the movable mirror 200a. Specifically, the movable mirror control unit 400b1 sends a mirror drive signal corresponding to the angle of the movable mirror 200a determined in step S22-1-2 to the movable mirror drive unit.

(Focus lens control process)
FIG. 15 is a flowchart showing the focus lens control process (step S22-2) of FIG.

In the first step S22-2-1, the optical center xyz coordinate calculation unit 400a1 calculates the z coordinate of the optical center OC of the optical element OE.

In the next step S22-2-2, the focus lens control unit 400b2 determines the position of the focus lens 200c in the optical axis direction based on the z coordinate of the optical center OC.

In the final step S22-2-3, the focus lens control unit 400b2 controls the focus lens 200c. Specifically, the focus lens control unit 400b2 sends a lens drive signal based on the position of the focus lens 200c in the optical axis direction determined in step S22-2-2 to the focus lens drive unit.

(Projection image control processing)
FIG. 16 is a flow chart showing the projection image control process (step S22-3) of FIG.

In the first step S22-3-1, the optical element tilt angle calculation unit 400a2 calculates the tilt angle (θx, θy) of the optical element OE.

In the next step S22-3-2, the projection image control unit 400b3 determines the brightness of the projection image based on the tilt angle (θx, θy) of the optical element OE.

In the final step S22-3-3, the projection image control unit 400b3 controls the projection image. Specifically, the projection image control unit 400b3 sends a light source drive signal modulated according to the brightness of the projection image determined in step S22-3-2 to the light source drive unit 100d. Note that the light source drive signal may be a signal modulated according to the brightness and shape correction of the projection image.

(Configuration Example 2 of Display Device Control)
FIG. 17 is a block diagram showing a second example of the control configuration of the display device 1. In FIG.

Configuration example 2 of the control of the display device 1 differs from configuration example 1 shown in FIG. 12 in that the acquisition unit 300 has an xy coordinate acquisition unit 300c and a z coordinate acquisition unit 300d instead of the xyz coordinate acquisition unit 300b, and the calculation unit 400a of the control unit 400 has an optical center xy coordinate calculation unit 400a11 and an optical center z coordinate calculation unit 400a12 instead of the xyz coordinate calculation unit 400a1.

(Overall control flow in configuration example 2 of the display device control)
Fig. 18 is a flowchart showing the flow of overall control in the configuration example 2 of Fig. 17. Fig. 18 is based on a processing algorithm executed by a CPU. The series of processes in Fig. 18 are started when the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates).

First, in step S31-1, the xy coordinate acquisition unit 300c performs an xy coordinate acquisition process and the z coordinate acquisition unit 300d performs a z coordinate acquisition process in parallel. Specifically, the xy coordinate acquisition unit 300c acquires the xy coordinates of at least three feature points FP from the images captured by the two cameras 300a, 300a that make up the stereo camera SC. The z coordinate acquisition unit 300d acquires the z coordinates of at least three feature points FP from the images captured by the two cameras 300a, 300a that make up the stereo camera SC.

Next, the control unit 400 performs the movable mirror control process of step S32-1, the focus lens control process of step S32-2, and the projection image control process of step S32-3 in parallel. Step S32-1 is the same as step S22-1 in FIG. 13. Step S32-2 is the same as step S22-2 in FIG. 13. Step S32-3 is the same as step S22-3 in FIG. 13.

Next, in step S33, the control unit 400 judges whether or not to end the processing each time step S32-1, step S32-2, and step S32-3 are executed. If the judgment here is positive, the flow ends, and if it is negative, the flow returns to steps S31-1 and S31-2, and the same processing is performed again. When the acquisition unit 300 is able to acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of camera 300a), the control unit 400 denies the judgment here (determines not to end the processing) and performs the processing of steps S31-1, S31-2, S32-1, S32-2, and S32-3 again. For example, when the user issues an end command or when the acquisition unit 300 is no longer able to acquire the three-dimensional coordinates of each feature point FP (when each feature point FP falls outside the angle of view of the camera 300a), the control unit 400 affirms the determination here (determines that processing should be ended) and ends the flow.

In the configuration example 2 of the control of the display device 1 described above, the control unit 400 controls the movable mirror 200a based on the two-dimensional position information (x, y) of each of the at least three feature points FP, controls the light source 100a based on the two-dimensional position information (x, y) and the distance information of the feature points FP (distance in the z-axis direction), and controls the focus lens 200c based on the distance information, in parallel.

Here, in general, it is relatively easy to obtain the x and y coordinates of each feature point FP, but obtaining the z coordinate requires restrictions on the device and arrangement used and a computation load. Furthermore, obtaining the x and y coordinates of the five axes of each feature point FP directly affects whether the observer can see the image or not, and has a higher priority than the remaining three parameters (z, θx, θy). On the other hand, the control of the projected image is limited by the frame rate of the image display, and is characterized by not being able to control at high speed compared to the movable mirror 200a. Therefore, in the above configuration example 2, as described above, the information acquisition period for obtaining the x and y coordinates of each feature point FP and the z coordinate, and the loop period of the subsequent calculation are independent. The process related to the control of the movable mirror 200a (first control loop) calculates the x and y coordinates at the highest speed and makes the projection optical axis follow the optical element OE (target). The process requiring the z coordinate (second control loop) updates information at a slower period than the process related to the control of the movable mirror 200a. The x and y coordinate information required to calculate θx and θy can be calculated using the latest information at the required timing. In this way, by dividing the control loop into two, the most important control of the movable mirror can be performed at high speed without being affected by the rate-limiting factors of the control speed.

<2. Display device according to second embodiment of the present technology>
Fig. 19 is a diagram showing a configuration example of a display device 2 according to a second embodiment of the present technology. As shown in Fig. 19, the display device 2 has a configuration substantially similar to that of the display device 1 according to the first embodiment, except that the optical axis (projection optical axis) of the projection optical system 200 and the optical axis of the camera 300a are coaxial.

In the display device 2, a half mirror 200d is disposed on the optical path of the image light IL between the focus lens 200c and the movable mirror 200a. The camera 300a is disposed at a position outside the optical path of the image light IL between the focus lens 200c and the movable mirror 200a, facing the half mirror 200d. In this case, the camera 300a is always in a state of capturing the position through which the projection optical axis passes within a predetermined angle of view (for example, a central angle of view). Therefore, if the projection optical axis deviates from the optical center OC of the optical element OE, the predetermined angle of view of the image captured by the camera 300a will also deviate from the optical center OC. Therefore, by moving the movable mirror 200a so as to cancel this deviation, the projection image can be always directed toward the optical center OC. Note that "the projection optical axis and the optical axis of the camera 300a are coaxial" does not necessarily mean that they are strictly coaxial, but means that deviation is allowed within a range in which the same action and effect is achieved.

FIG. 20 is a flowchart showing the overall control flow of an example configuration of a display device 2 according to a second embodiment of the present technology. The flowchart in FIG. 20 is based on a processing algorithm executed by a CPU. The series of processes in FIG. 20 are started when the camera 300a captures coordinates (when the acquisition unit 300 acquires coordinates).

In the first step S11, the acquisition unit 300 acquires the two-dimensional coordinates (x, y) of each of at least three feature points FP.

In the next step S12, the calculation unit 400a of the control unit 400 calculates the two-dimensional coordinates (x, y) of the optical center OC of the optical element OE based on the two-dimensional coordinates (x, y) of each of the at least three feature points FP.

In the next step S13, the control unit 400 determines whether or not the projection optical axis passes through the optical center OC of the optical element OE. Specifically, the control unit 400 determines whether or not there is a misalignment between the projection optical axis and the optical center OC based on the position (two-dimensional coordinates) of the optical center OC in the image captured by the camera 300a. If the determination here is positive (if it is determined that there is no misalignment), the process returns to step S11, and if it is negative (if it is determined that there is a misalignment), the process proceeds to step S14.

In the next step S14, the control unit 400 controls the movable mirror 200a so that the projection optical axis passes through the optical center OC of the optical element OE.

In the final step S15, the control unit 400 determines whether or not to end the process. For example, when the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of the camera 300a), the determination here is negative (determined not to end the process) and the process returns to step S11. On the other hand, for example, when the user issues an end command or the acquisition unit 300a can no longer acquire the three-dimensional coordinates of each feature point FP (when each feature point FP falls outside the angle of view of the camera 300a), the determination here is positive (determined to end the process) and the flow ends.

In the display device 2 described above, the camera 300a is arranged to recognize the vicinity of the eyepiece optical system 20 through the movable mirror 200a, so that it is possible to achieve both wide-range and high-precision recognition. The imaging range of the camera 300a can cover the entire range within which the light projection system 10 can project. In addition, since the origins of the light projection coordinate system and the camera coordinate system coincide, there is also the advantage that the initial calibration can be simplified and positional deviation errors due to each perturbation can be suppressed. However, in this case, since the recognition is via the movable mirror 200a, the detected coordinates are relative coordinates rather than absolute coordinates. Therefore, when the observer is lost, it becomes unclear in which direction the movable mirror 200a should be pointed. This can be solved by installing a camera for absolute coordinate recognition. This camera does not require high performance in terms of resolution and shooting speed, so a relatively inexpensive one may be prepared, or an area that is not coaxial with the projection optical axis may be provided in part of the imaging field of the camera 300a and used by cutting out that area. Another solution is to provide two types of control methods for the movable mirror 200a: a tracking mode and a search mode. When the position of the observer is unknown, the search mode is used, and the observer is searched for by two-dimensionally scanning the movable range of the movable mirror 200a. When the position of the observer is identified, the mode is switched to the tracking mode and control is performed using the above-mentioned method. The at least one camera 300a may be any type, such as a visible light camera, an infrared camera, a stereo camera, or a TOF camera, and it is also possible to combine it with the display device 1 according to the first embodiment.

<3. Display device according to a third embodiment of the present technology>
21 is a diagram showing a configuration example of a display device 3 according to a third embodiment of the present technology. The display device 3 has a similar configuration to the display device 1 according to the first embodiment, except that an optical element OE and at least three feature points FP are provided on a spectacle lens GL.

The eyeglass lens GL may or may not be a component of the eyepiece optical system 20. If the optical element OE is provided on the eyeglass lens GL, there is a possibility that the pupil of the eyeball EB and the optical center OC of the optical element OE will not coincide, which may result in disadvantages such as the need for greater precision and measures to address changes over time, or the need to additionally detect and track the position of the pupil, but it also has advantages such as a high degree of freedom in designing the feature points FP, improved detection accuracy of the feature points FP, easy manufacture, high safety, and low barriers to wearing. Here, a HOE can be mainly used for the optical element OE, but a diffractive optical element (DOE) or the like can also be used.

In the display device 3, as an example, as shown in FIG. 21, at least three feature points FP are provided in the area around the optical element OE of the eyeglass lens GL fitted into the rim (frame) of the eyeglass frame GF. Note that in the display device 3, the eyeglass frame GF may be a rimless frame.

<4. Display device according to fourth embodiment of the present technology>
22 is a diagram showing a configuration example of a display device 4 according to a fourth embodiment of the present technology. The display device 4 has a similar configuration to the display device 3 according to the third embodiment, except that an optical element OE is provided on a spectacle lens GL and at least three feature points FP are provided on a spectacle frame GF.

In the display device 4, as an example, at least three feature points FP are provided on the rim of the eyeglass frame GF, as shown in FIG. 22.

<5. Display device according to a fifth embodiment of the present technology>
23 is a diagram showing a configuration example of a display device 5 according to a fifth embodiment of the present technology. The display device 5 has a similar configuration to the display device 3 according to the third embodiment, except that the optical element OE and parts of the at least three feature points FP are provided on the eyeglass lens GL, and the other parts of the at least three feature points FP are provided on the eyeglass frame GF.

In the display device 5, as an example, as shown in FIG. 23, an optical element OE and at least one feature point FP are provided on the eyeglass lens GL, and at least two feature points FP are provided on the eyeglass frame GF. Note that in the display device 5, the optical element OE and at least two feature points FP may be provided on the eyeglass lens GL, and at least one feature point FP may be provided on the eyeglass frame GF.

<6. Display device according to a sixth embodiment of the present technology>
25 is a diagram showing a configuration example of a display device 6 according to a sixth embodiment of the present technology. As shown in FIG. 25, the display device 6 has a configuration similar to that of the display device 1 according to the first embodiment, except that the acquisition unit 300 has an infrared light source 300e (e.g., an LED, a laser, etc.) that irradiates infrared light to at least three feature points FP near the focus lens 200c, and has an infrared camera (a camera sensitive to infrared wavelengths) as the camera 300a. Note that the infrared light source 500 may be a part of the light source 100a, or may be disposed on the optical path of the light from the light source 100a.

In the display device 6, a visible light cut filter may be provided on the camera 300a as a way to reduce the amount of processing. Here, markers made of a retroreflective material are used as each feature point FP. As shown in FIG. 26, markers made of a retroreflective material have the property that the incident light and reflected light are in the same direction, unlike specular reflection (regular reflection) on a plane mirror. By utilizing this property, in principle, even if the observer moves significantly, strong light reflected by the marker will always return to the periphery of the projection optical system 200 (more specifically, the infrared light source 300e), so the position of the marker can be detected with high accuracy. In addition, since the display device 6 has the infrared light source 300e as an illumination light source, the position of the marker can be detected even if the observer is in a dark environment.

(TOF camera)
FIG. 27 is a block diagram showing an example of the configuration of a TOF (Time of Flight) camera as the camera 300a.

As shown in FIG. 27, the TOF camera serving as camera 300a has a light emitting unit 300a1, a light receiving unit 300a2, and a distance calculation unit 300a3. The measurement method of the TOF camera may be a direct TOF method or an indirect TOF method.

The TOF camera can be used, for example, to obtain the z coordinate (distance) of each feature point FP, and can be used in conjunction with a visible light camera or an infrared camera to obtain the xy coordinates of the feature points FP. If the TOF camera has high xy resolution, it can obtain all xyz coordinates, which is expected to reduce the number of parts, making it smaller, lighter, and more cost-effective.

(Event Camera)
FIG. 28 is a block diagram showing an example of the configuration of a camera 300a (event camera) having an event sensor.

The event sensor of the event camera as camera 300a has a luminance change detection unit 300a4 that detects luminance changes in each pixel, and a luminance change data output unit 300a5 that outputs only data related to pixels whose luminance has changed, as shown in FIG. 28. The event sensor is also called an event-based vision sensor (EVS).

An event sensor is a sensor that detects changes in the brightness of each pixel asynchronously and outputs only the changed data, combining it with coordinate and time information, achieving high-speed, low-latency data output. Event cameras equipped with event sensors can capture changes in feature points faster than normal cameras while reducing the amount of information they handle, enabling a more natural tracking experience and at the same time reducing the processing load in subsequent stages, contributing to smaller, lighter models and lower power consumption.

7. Effects of the display device according to the present technology
The display device according to the present technology described above (for example, any of the display devices 1 to 6 according to the first to sixth embodiments) comprises an optical projection system 10 including a light source 100a, and an eyepiece optical system 20 that guides light projected from the optical projection system 10 to an eyeball EB, where the eyepiece optical system 20 includes an optical element OE and three or more feature points FP whose relative positions with respect to the optical element OE are fixed, and the optical projection system 10 includes an acquisition unit 300 that acquires at least two-dimensional position information of each of at least three of the three or more feature points FP.

The display device according to this technology can accurately detect at least two-dimensional information for each of the at least three feature points FP.

As a result, the display device according to the present technology can provide a display device capable of detecting the relative position between the light projection system 10 and the eyepiece optical system 20 with high accuracy. The display device according to the present technology can detect the relative position with high accuracy, and therefore can realize a display device with excellent image tracking performance with respect to the eyepiece optical system 20 worn by the observer. The display device according to the present technology has excellent image tracking performance even in a usage mode in which the separate light projection system and eyepiece optical system are arranged at a considerable spatial distance (for example, farther apart than when both the light projection system and the eyepiece optical system are worn by the observer) and the positional relationship between the two changes dynamically in real time.

The display method according to the present technology (for example, a display method performed using any of the display devices 1 to 6 according to the first to sixth embodiments) includes a step of acquiring at least two-dimensional position information of at least three of three or more feature points FP whose relative positions with respect to an optical element OE that guides light projected from an optical projection system 10 to an eyeball EB are fixed, and a step of controlling the optical projection system 10 based on the acquisition results from the acquisition step.

The display method according to the present technology can accurately detect at least two-dimensional information for each of the at least three feature points FP.

As a result, the display method according to the present technology can provide a display method capable of detecting the relative position between the light projection system 10 and the eyepiece optical system 20 with high accuracy. The display method according to the present technology can detect the relative position with high accuracy, and therefore can realize a display method with excellent image tracking for the eyepiece optical system 20 worn by the observer. The display method according to the present technology has excellent image tracking even in a usage mode in which the separate light projection system and eyepiece optical system are arranged at a considerable spatial distance (for example, farther apart than when both the light projection system and the eyepiece optical system are worn by the observer) and the positional relationship between the two changes dynamically in real time.

8. Modifications of the present technology
The display device and the display method using the display device according to each embodiment of the present technology described above can be modified as appropriate.

For example, in the display device according to each of the above embodiments, the acquisition unit 300 may acquire only two-dimensional position information for each of at least three feature points FP. Even in this case, it is possible to draw an image on the eyeball EB of the user who is the observer by controlling the movable mirror 200a based on the two-dimensional position information.

For example, in the display device according to each of the above embodiments, the acquisition unit 300 may acquire at least two-dimensional position information (e.g., two-dimensional position information only, or two-dimensional position information and distance information) for each of the at least four feature points FP. In this case, the relative position between the optical projection system 10 and the eyepiece optical system 20 can be determined with greater accuracy.

For example, in the display devices according to the above embodiments, the movable deflection element (e.g., the movable mirror 200a) is controlled based on two-dimensional coordinates (x, y), but the movable deflection element may also be controlled based on three-dimensional coordinates (x, y, z).

For example, in the display device according to each of the above embodiments, the projection optical system 200 does not need to have a focus lens 200c and a focus lens driving unit.

For example, in the display device according to each of the above embodiments, the control unit 400 does not need to have a focus lens control unit and/or a projection image control unit.

For example, in the display device according to each of the above embodiments, only one of focus lens position control and projected image brightness control may be performed.

For example, in the display device according to each of the above embodiments, the control unit 400 may control only one of the light source 100a and the projection optical system 200.

For example, in the display devices according to the above embodiments, the image light generating unit is of the scanning type, but it may also be of the non-scanning type, including, for example, a liquid crystal display.

For example, in the display devices according to the above embodiments, the optical projection system 10 can be worn on the head of the user who is the observer.

At least a portion of the configuration of the eyeball information detection device in each of the above embodiments may be combined to the extent that they are not mutually inconsistent.

The present technology can also be configured as follows.
(1) a light projection system including a light source;
an eyepiece optical system that guides the light projected from the light projection system to an eyeball;
Equipped with
The eyepiece optical system includes:
An optical element;
Three or more feature points having fixed relative positions with respect to the optical element;
Including,
The display device is provided with an acquisition unit that acquires at least two-dimensional position information of at least three of the three or more feature points.
(2) The display device according to (1), wherein the at least three feature points are not on the same line.
(3) The display device according to (1) or (2), wherein the optical element is provided in a contact lens that is attached to the eyeball.
(4) The display device according to (3), wherein the at least three feature points are provided on the contact lens.
(5) The display device according to (4), wherein the acquisition unit is capable of acquiring at least the two-dimensional position information of each of the at least three feature points even when the orientation of the eyeball changes.
(6) The display device according to (1) or (2), wherein the optical element is provided on a lens attached to an eyeglass frame.
(7) The display device described in (6), wherein all of the at least three feature points are provided on the lens or the eyeglass frame, or some of the at least three feature points are provided on the lens and others are provided on the eyeglass frame.
(8) The display device according to any one of (1) to (7), wherein the at least three characteristic points and the optical center of the optical element are on the same plane.
(9) The display device according to any one of (1) to (8), wherein the acquisition unit has at least one camera.
(10) The display device according to (9), wherein the at least one camera includes a visible light camera and/or a Time-of-Flight camera.
(11) The display device according to (9) or (10), wherein the at least one camera includes a plurality of cameras constituting a stereo camera.
(12) The display device according to any one of (1) to (11), wherein the optical projection system includes the acquisition unit.
(13) A display device described in any one of (9) to (12), wherein the acquisition unit has an infrared light source, the camera is sensitive to infrared wavelengths, and the at least three feature points are made of a retroreflective material.
(14) The optical projection system includes:
a projection optical system disposed on an optical path of light from the light source;
The display device according to any one of (9) to (13), wherein an optical axis of the projection optical system and an optical axis of the camera are coaxial.
(15) The optical projection system includes:
a projection optical system disposed on an optical path of light from the light source;
a control unit that controls the light source and/or the projection optical system based on the result of the acquisition by the acquisition unit;
The display device according to any one of (9) to (14),
(16) The projection optical system includes a movable deflection element and a focus lens,
The acquisition unit is
Two-dimensional position information of each of the at least three feature points in a plane perpendicular to the optical axis direction of the camera;
Distance information between the camera and each of the at least three feature points in a direction of an optical axis of the camera;
Get
The control unit is
controlling the movable deflection element based on at least the two-dimensional position information;
controlling the light source based on the two-dimensional position information and the distance information, and/or controlling the focus lens based on the distance information;
The display device according to (15), wherein the above steps are performed in parallel.
(17) The display device according to any one of (1) to (16), wherein at least two of the at least three characteristic points are integrally provided.
(18) The display device according to any one of (1) to (17), wherein the optical projection system and the eyepiece optical system are separate entities.
(19) acquiring at least two-dimensional position information of at least three of the three or more feature points whose relative positions with respect to an optical element that guides light projected from a light projection system to an eyeball are fixed;
controlling a part of the optical projection system based on the results of the acquiring step;
Including, how to display.
(20) The optical projection system includes a light source, a movable deflection element, and a focus lens;
The step of acquiring at least two-dimensional position information includes:
acquiring two-dimensional position information of each of the at least three feature points in a plane perpendicular to an optical axis direction of the camera;
acquiring distance information between the camera and each of the at least three feature points in a direction of an optical axis of the camera;
Including,
The controlling step includes:
controlling the movable deflection element based on at least the two-dimensional position information;
controlling the light source based on the two-dimensional position information and the distance information, and/or controlling the focus lens based on the distance information;
The display method according to (19),

1, 2, 3, 4, 5, 6: Display device 10: Light projection system 20: Eyepiece optical system 100a: Light source 200: Projection optical system 200a: Movable mirror (movable deflection element)
200c: Focus lens 300: Acquisition unit 300a: Camera 300e: Infrared light source 400: Control unit EB: Eyeball FP: Feature point OE: Optical element OC: Optical center CL: Contact lens GL: Glasses lens (lens)
GF: Glasses frame IL: Image light (light)

Claims (20)

a light projection system including a light source;
an eyepiece optical system that guides the light projected from the light projection system to an eyeball;
Equipped with
The eyepiece optical system includes:
An optical element;
Three or more feature points having fixed relative positions with respect to the optical element;
Including,
The display device is provided with an acquisition unit that acquires at least two-dimensional position information of at least three of the three or more feature points. The display device of claim 1, wherein the at least three feature points are not collinear. The display device according to claim 1, wherein the optical element is provided in a contact lens that is attached to the eyeball. The display device of claim 3, wherein the at least three feature points are provided on the contact lens. The display device according to claim 4, wherein the acquisition unit is capable of acquiring at least the two-dimensional position information of each of the at least three feature points even if the orientation of the eyeball changes. The display device according to claim 1, wherein the optical element is provided on a lens attached to an eyeglass frame. The display device according to claim 6, wherein all of the at least three feature points are provided on the lens or the eyeglass frame, or some of the at least three feature points are provided on the lens and others are provided on the eyeglass frame. The display device of claim 1, wherein the at least three feature points and the optical center of the optical element are on the same plane. The display device according to claim 1, wherein the acquisition unit has at least one camera. The display device of claim 9, wherein the at least one camera includes a visible light camera and/or a time-of-flight camera. The display device according to claim 9, wherein the at least one camera includes a plurality of cameras constituting a stereo camera. The display device of claim 1, wherein the optical projection system includes the acquisition unit. The acquisition unit has an infrared light source,
the camera is sensitive to infrared wavelengths;
The display device of claim 9 , wherein the at least three features are made of retroreflective material. The light projection system includes:
a projection optical system disposed on an optical path of light from the light source;
10. The display device according to claim 9, wherein an optical axis of the projection optical system and an optical axis of the camera are coaxial. The light projection system includes:
a projection optical system disposed on an optical path of light from the light source;
a control unit that controls the light source and/or the projection optical system based on the result of the acquisition by the acquisition unit;
The display device according to claim 9 , the projection optical system includes a movable deflection element and a focus lens;
The acquisition unit is
Two-dimensional position information of each of the at least three feature points in a plane perpendicular to the optical axis direction of the camera;
Distance information between the camera and each of the at least three feature points in a direction of an optical axis of the camera;
Get
The control unit is
controlling the movable deflection element based on at least the two-dimensional position information;
controlling the light source based on the two-dimensional position information and the distance information, and/or controlling the focus lens based on the distance information;
The display device according to claim 15 , wherein the above steps are performed in parallel. The display device according to claim 1, wherein at least two of the at least three feature points are integrally provided. The display device according to claim 1, wherein the optical projection system and the eyepiece optical system are separate. acquiring at least two-dimensional position information of at least three of the three or more feature points whose relative positions with respect to an optical element that guides light projected from a light projection system to the eyeball are fixed;
controlling a part of the optical projection system based on the results of the acquiring step;
Including, how to display. the light projection system includes a light source, a movable deflection element, and a focus lens;
The step of acquiring at least two-dimensional position information includes:
acquiring two-dimensional position information of each of the at least three feature points in a plane perpendicular to an optical axis direction of the camera;
acquiring distance information between the camera and each of the at least three feature points in a direction of an optical axis of the camera;
Including,
The controlling step includes:
controlling the movable deflection element based on at least the two-dimensional position information;
controlling the light source based on the two-dimensional position information and the distance information, and/or controlling the focus lens based on the distance information;
20. The display method of claim 19, comprising: