JP2018160735A - Transmission type display apparatus, display control method, and computer program - Google Patents

Abstract

In a transmissive display device, the visibility of a pointer is improved. A transmissive display device having a light transmission property, a display unit that displays a target image to be displayed and a pointer image so as to be superimposed on an external world that is visible through the self, and a pointer image A display control unit that controls a display mode of the image capturing unit, and an imaging unit that captures an image of the outside world, and the display control unit includes an external world image obtained by imaging, a target image, an external image and a composite image of the target image, A transmissive display device that uses any one of them as a reference image and changes the display mode of the pointer image according to the feature amount of the region including the display position of the pointer image in the reference image. [Selection] Figure 8

Description

  The present invention relates to a transmissive display device.

  As a head-mounted display device (Head Mounted Display (HMD)) that is mounted on the head and displays an image or the like within the visual field of the user, a transmission type that can visually see the scenery of the outside world together with the image when worn There are known head-mounted display devices. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and guides the generated image light to a user's eye using a projection optical system, a light guide plate, and the like. This makes the user recognize a virtual image. The user controls the head-mounted display device by operating a pointer such as a mouse cursor displayed on the liquid crystal display. However, depending on the scenery of the outside world and the colors and patterns in the display image, the pointer may be difficult to see and may lose sight of the pointer. In Patent Document 1, in a transmissive display device, when a display image is displayed while being superimposed on an external landscape image, the display image is displayed based on the luminance of the entire external landscape image and the entire display image. A technique for calculating the luminance and controlling the color or color density for displaying the display image in accordance with the display luminance is disclosed.

JP 2006-142988 A

  However, in the technique described in Patent Document 1, for example, when the brightness of the display image is significantly lower than the brightness of the outside scene, or when the outside scene rich in color is perceived through the outside, The color of the landscape affects the color of the pointer, which may cause a problem that the user cannot visually recognize the pointer or it is difficult to visually recognize the pointer. Such a problem is not limited to a transmissive head-mounted display device, but is common to a transmissive display device that displays an image or the like superimposed on an external landscape. Therefore, a technique for improving the visibility of a pointer in a transmissive display device is desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one embodiment of the present invention, a transmissive display device is provided. This transmissive display device has a light-transmitting property, and displays a target image to be displayed and a pointer image so as to be superimposed on an external world that is visible through the display device; a display mode of the pointer image A display control unit for controlling; an imaging unit for imaging the outside world; and the display control unit, an external world image obtained by the imaging, the target image, the external image, and a composite image of the target image, The reference image is used as a reference image, and the display mode of the pointer image is changed and displayed according to the feature amount of the region including the display position of the pointer image in the reference image.
According to the transmissive display device of this aspect, the pointer image in the reference image using any one of the external image obtained by imaging, the target image, and the composite image of the external image and the target image as the reference image. Since the display mode of the pointer image is changed according to the feature amount of the area including the display position, the pointer image can be displayed in a display mode that takes into account the characteristics of the reference image. Can be improved.

  (2) In the transmissive display device of the above aspect, the reference image may be the composite image. According to this aspect of the transmissive display device, since the reference image is a composite image, the display mode of the pointer image can be controlled in consideration of the feature quantities of the external image and the target image. Therefore, the visibility of the pointer image can be improved when the target image is displayed in a superimposed manner on the external world that is visible through the screen, and the pointer image is displayed in a superimposed manner.

  (3) The transmissive display device according to the above aspect further includes an average hue value calculation unit that calculates an average hue value of a region including the display position of the pointer image in the reference image; The pointer image may be displayed with the color of the pointer image complementary to the average hue value. According to this aspect of the transmissive display device, since the pointer image is displayed with the color of the pointer image complementary to the average hue value, the pointer image is displayed more than the configuration in which the color of the pointer image is not complementary to the average hue value. Easy to see.

  (4) In the transmissive display device according to the above aspect, the display device may further include an edge tendency calculating unit that calculates an edge tendency of a region including the display position of the pointer image in the reference image; The pointer image may be displayed with a pattern having a directionality determined based on the edge tendency. According to this aspect of the transmissive display device, the pointer image is displayed with a pattern having a direction determined based on the edge tendency, so that the pointer image is highly visible and the pointer image itself is displayed. Visibility can be improved.

  (5) In the transmissive display device according to the above aspect, the display control unit may display the pointer image in a striped pattern in which the pattern is repeated in a direction intersecting with a composite vector of the edge tendency. According to this aspect of the transmissive display device, the pointer image is displayed in a striped pattern that is repeated in a direction intersecting the composite vector of the edge tendency, so that the visibility of the striped pattern of the pointer image is improved. The visibility of the pointer image itself can be improved.

  (6) The transmissive display device according to the above aspect further includes an average hue value calculation unit that calculates an average hue value of a region including the display position of the pointer image in the reference image; A pointer image may be displayed as a solid with parallax, and the amount of change in parallax may be set based on the average hue value. According to this aspect of the transmissive display device, the pointer image is displayed as a solid with parallax, and the amount of change in parallax is set based on the average hue value, so the pointer image has moved to an area with a different average hue value In this case, the visibility of the pointer image can be improved, and the followability to the pointer image (performance to follow with the eyes) can be improved.

  (7) In the transmissive display device according to the above aspect, the display device further includes an average hue value calculation unit that calculates an average hue value of a region including the display position of the pointer image in the reference image; By alternately switching between displaying the pointer image as a solid with parallax and displaying the pointer image as a non-parallax surface, and setting the period for applying the parallax based on the average hue value Also good. According to this aspect of the transmissive display device, the pointer image is displayed alternately as a solid with a parallax and the pointer image is displayed as a non-parallax surface, and the display is based on the average hue value. Since the interval to add parallax is set, the pointer image is made conspicuous to improve visibility, and the pointer image is made to stand out by changing the cycle to add parallax even when the pointer image moves to an area with a different average hue value And followability to the pointer image can be improved.

  (8) In the transmissive display device according to the above aspect, the display control unit displays the pointer image as a solid with a parallax; and displays the focal position of the pointer image displayed as the solid from the display unit. You may perform alternately moving a predetermined distance in the direction which goes to an external world, and moving the said predetermined distance in the direction which goes to the said display part from the said external world. According to this aspect of the transmissive display device, the pointer image is displayed as a solid with parallax, and the focal position of the pointer image displayed as a solid is moved by a predetermined distance in a direction from the display unit toward the outside world. Since the predetermined distance in the direction from the outside world toward the display unit is alternately performed, the pointer image displayed as a three-dimensional object is displayed in a concave manner and is displayed in a protruding manner alternately. It is possible to improve the visibility of the pointer image.

  (9) In the transmissive display device according to the above aspect, the display control unit may periodically change the brightness of at least a part of the pointer image to display the pointer image. According to the transmissive display device of this aspect, since the pointer image is displayed by periodically changing the brightness of at least a part of the pointer image, a part of the pointer image can be blinked and displayed. The visibility of the image can be further improved.

  (10) In the transmissive display device according to the above aspect, the display control unit may control the display mode of the pointer image when triggered by detection of a predetermined operation in the transmissive display device. According to the transmissive display device of this aspect, since the control of the display mode of the pointer image is executed in response to detection of the predetermined operation in the transmissive display device, the trigger of executing the control of the display mode of the pointer image is the transmission type. The user of the display device can make adjustments, and user convenience can be improved.

  The present invention can be realized in various forms. For example, the present invention can be realized in the form of a display control method in a transmissive display device, a computer program for realizing the display control method, a recording medium on which the computer program is recorded, and the like.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus as 1st Embodiment of this invention. It is a principal part top view which shows the structure of the optical system with which an image display part is provided. It is a figure which shows the principal part structure of the image display part seen from the user. It is a figure for demonstrating the angle of view of a camera. It is a block diagram which shows the structure of HMD functionally. It is a block diagram which shows the structure of a control apparatus functionally. It is explanatory drawing which shows an example of the augmented reality display by HMD. It is a flowchart which shows the process sequence of a pointer image display control process. It is a flowchart which shows the detailed process sequence of a color tone change process (step S110). It is a flowchart which shows the detailed process sequence of a reference | standard image setting process (step S200). It is explanatory drawing which shows typically an example of the pointer image after execution of a color tone change process (step S110). It is a flowchart which shows the detailed process sequence of direction change process (step S120). It is explanatory drawing which shows typically an example of the pointer image after execution of direction change processing (step S120). It is a flowchart which shows the detailed process sequence of a parallax change process (step S130). It is explanatory drawing which shows typically an example of the pointer image after execution of a parallax change process (step S130). It is a flowchart which shows the detailed process sequence of the parallax change process (step S130a) in 2nd Embodiment. It is a flowchart which shows the detailed process sequence of the parallax change process (step S130b) in 3rd Embodiment. It is explanatory drawing which shows the pattern of the pointer image in the modification 3 typically.

A. First embodiment:
A1. Overall configuration of the transmissive display device:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device 100 as a first embodiment of the present invention. The head-mounted display device 100 is a display device that is mounted on a user's head, and is also referred to as a head-mounted display (HMD). The HMD 100 is a see-through type (transmission type) head-mounted display device in which an image floats in an external environment visually recognized through a glass.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize an image, and a control device (controller) 10 that controls the image display unit 20.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate on a support having a right holding unit 21, a left holding unit 23, and a front frame 27. 28. In the present embodiment, the image display unit 20 corresponds to a display unit in a means for solving the problem.

  Each of the right holding unit 21 and the left holding unit 23 extends rearward from both end portions of the front frame 27 and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is worn. The left holding part 23 is provided to extend from the end EL of the front frame 27 to a position corresponding to the left side of the user when the image display part 20 is worn.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other. This connection position corresponds to the position between the eyebrows of the user when the image display unit 20 is mounted. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt that is in contact with the back of the user's head when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 can be firmly held on the user's head by the belt.

  The right display unit 22 displays an image by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user when the image display unit 20 is worn. The left display unit 24 displays an image by the left light guide plate 28. The left display unit 24 is provided in the left holding unit 23 and is located in the vicinity of the user's left head when the image display unit 20 is worn.

  The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units (for example, prisms) formed of a light transmissive resin or the like, and use image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the person. A dimming plate may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is a thin plate-like optical element having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the surface of the front frame 27 (the surface opposite to the surface facing the user's eyes). By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively. Along with the landscape, the user visually recognizes an image (an augmented reality (AR) image) (this is also referred to as “displaying an image”). When external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, image light constituting the image and external light are incident on the user's eyes. To do. For this reason, the visibility of the image for the user is affected by the intensity of external light.

  For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, it is possible to adjust the ease of visual recognition of the image. In a typical example, it is possible to select a dimming plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. Moreover, sunlight can be suppressed and the visibility of an image can be improved. In addition, when the dimming plate is used, it is possible to protect the right light guide plate 26 and the left light guide plate 28 and to suppress the damage to the right light guide plate 26 and the left light guide plate 28 and the adhesion of dirt. The light control plate may be detachable from the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28. In addition, a plurality of types of light control plates may be exchanged and removable, or the light control plates may be omitted.

  The camera 61 is disposed on the front frame 27 of the image display unit 20. The camera 61 is provided on the front surface of the front frame 27 at a position that does not block outside light that passes through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27. The camera 61 may be disposed on the end EL side of the front frame 27 or may be disposed at a connection portion between the right light guide plate 26 and the left light guide plate 28.

  The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of this embodiment is a monocular camera, but a stereo camera may be adopted. The camera 61 images at least a part of the outside world (real space) in the front side direction of the HMD 100, in other words, in the viewing direction that the user visually recognizes when the image display unit 20 is mounted. In other words, the camera 61 images a range or direction that overlaps the user's field of view, and images a direction that the user visually recognizes. The angle of view of the camera 61 can be set as appropriate. In the present embodiment, the width of the angle of view of the camera 61 is set so that the entire field of view of the user that the user can see through the right light guide plate 26 and the left light guide plate 28 is imaged. The camera 61 performs imaging according to the control of the control function unit 150 (FIG. 6), and outputs the obtained imaging data to the control function unit 150. In the present embodiment, the camera 61 corresponds to an imaging unit in a means for solving the problem.

  The HMD 100 may include a distance measuring sensor that detects a distance to a measurement object positioned in a preset measurement direction. The distance measuring sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 of the front frame 27. The measurement direction of the distance measuring sensor can be the front side direction of the HMD 100 (the direction overlapping the imaging direction of the camera 61). The distance measuring sensor can be composed of, for example, a light emitting unit such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured by the light source. In this case, the distance is obtained by triangular distance measurement processing or distance measurement processing based on a time difference. The distance measuring sensor may be configured by, for example, a transmission unit that emits ultrasonic waves and a reception unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance is obtained by distance measurement processing based on the time difference. The distance measuring sensor measures the distance according to the instruction from the control function unit 150 and outputs the detection result to the control function unit 150, similarly to the camera 61.

  FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. For convenience of explanation, FIG. 2 shows a user's right eye RE and left eye LE. As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically.

  As a configuration for allowing the right eye RE to visually recognize an image (AR image), the right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 and a right optical system 251. The OLED unit 221 emits image light. The right optical system 251 includes a lens group and the like, and guides the image light L emitted from the OLED unit 221 to the right light guide plate 26.

  The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by light emitting elements that emit light by organic electroluminescence and emit color lights of R (red), G (green), and B (blue). In the OLED panel 223, a plurality of pixels each having a unit including one R, G, and B element as one pixel are arranged in a matrix.

  The OLED drive circuit 225 performs selection and energization of the light emitting elements included in the OLED panel 223 under the control of the control function unit 150 (FIG. 6) described later, and causes the light emitting elements to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be configured by a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate that is fixed to the back surface of the OLED panel 223. A temperature sensor 217 (FIG. 5) described later is mounted on this board. Note that the OLED panel 223 may employ a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to the colors R, G, and B are stacked. Further, an OLED panel 223 having a WRGB configuration including a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be employed.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and then emitted from the right light guide plate 26 to the right eye RE. The image light L forms an image with the retina of the right eye RE, thereby allowing the user to visually recognize the image.

  As a configuration for causing the left eye LE to visually recognize an image (AR image), the left display unit 24 includes an OLED unit 241 and a left optical system 252. The OLED unit 241 emits image light. The left optical system 252 includes a lens group and the like, and guides the image light L emitted from the OLED unit 241 to the left light guide plate 28. The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. Details of each part are the same as those of the OLED unit 221, the OLED panel 223, and the OLED drive circuit 225. A temperature sensor 239 (FIG. 5) is mounted on the substrate fixed to the back surface of the OLED panel 243. The details of the left optical system 252 are the same as those of the right optical system 251 described above.

  According to the configuration described above, the HMD 100 can function as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 enter the right eye RE of the user. The image light L reflected by the half mirror 281 and the external light OL transmitted through the left light guide plate 28 enter the left eye LE of the user. As described above, the HMD 100 causes the image light L of the image processed inside and the external light OL to overlap and enter the user's eyes. As a result, the user can see the outside landscape (real world) through the right light guide plate 26 and the left light guide plate 28, and the virtual image (virtual image image, AR image) by the image light L so as to overlap the outside world. Is visible.

  The right optical system 251 and the right light guide plate 26 are collectively referred to as “right light guide”, and the left optical system 252 and the left light guide plate 28 are also collectively referred to as “left light guide”. The configurations of the right light guide unit and the left light guide unit are not limited to the above-described examples, and any method can be used as long as an image is formed in front of the user's eyes using image light. For example, a diffraction grating or a transflective film may be used for the right light guide and the left light guide.

  In FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector provided in the lower part of the control device 10, and is connected to various circuits inside the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable or an optical fiber cable that transmits digital data. The connection cable 40 may further include a metal cable that transmits analog data. A connector 46 is provided in the middle of the connection cable 40.

  The connector 46 is a jack for connecting a stereo mini-plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the example of the present embodiment shown in FIG. 1, a right earphone 32 and a left earphone 34 constituting stereo headphones and a headset 30 having a microphone 63 are connected to the connector 46.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the line of sight of the user. The microphone 63 collects sound and outputs a sound signal to the sound interface 182 (FIG. 5). The microphone 63 may be a monaural microphone or a stereo microphone, and may be a directional microphone or an omnidirectional microphone.

  The control device 10 is a device for controlling the HMD 100. The control device 10 includes a lighting unit 12, a track pad 14, a direction key 16, an enter key 17, and a power switch 18. The lighting unit 12 notifies the operation state of the HMD 100 (for example, ON / OFF of the power supply) by its light emission mode. As the lighting unit 12, for example, an LED (Light Emitting Diode) can be used.

  The track pad 14 detects a contact operation on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The determination key 17 detects a pressing operation and outputs a signal for determining the content operated in the control device 10. The power switch 18 switches the power state of the HMD 100 by detecting a slide operation of the switch.

  FIG. 3 is a diagram illustrating a main configuration of the image display unit 20 as viewed from the user. In FIG. 3, the connection cable 40, the right earphone 32, and the left earphone 34 are not shown. In the state of FIG. 3, the back sides of the right light guide plate 26 and the left light guide plate 28 can be visually recognized, the half mirror 261 for irradiating the right eye RE with image light, and the image light for irradiating the left eye LE with image light. The half mirror 281 can be visually recognized as a substantially rectangular area. The user permeates through the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 to visually recognize the scenery of the outside world and visually recognizes a rectangular display image at the position of the half mirrors 261 and 281. .

  FIG. 4 is a diagram for explaining the angle of view of the camera 61. In FIG. 4, the camera 61 and the user's right eye RE and left eye LE are schematically shown in plan view, and the angle of view (imaging range) of the camera 61 is indicated by θ. Note that the angle of view θ of the camera 61 extends in the horizontal direction as shown in the figure, and also in the vertical direction as in a general digital camera.

  As described above, the camera 61 is arranged at the right end of the image display unit 20 and images the direction of the user's line of sight (that is, the front of the user). For this reason, the optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The scenery of the outside world that can be visually recognized with the user wearing the HMD 100 is not always at infinity. For example, when the user gazes at the object OB with both eyes, the user's line of sight is directed toward the object OB as indicated by reference numerals RD and LD in the figure. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often 1 m to 4 m. Therefore, for the HMD 100, an upper limit and a lower limit of the distance from the user to the object OB during normal use may be determined. This standard may be obtained in advance and preset in the HMD 100, or may be set by the user. The optical axis and the angle of view of the camera 61 are set so that the object OB is included in the angle of view when the distance to the object OB during normal use corresponds to the set upper and lower limits. It is preferred that

  In general, a human viewing angle is approximately 200 degrees in the horizontal direction and approximately 125 degrees in the vertical direction. Among them, the effective visual field with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. A stable gaze field in which a gaze point that a person gazes at appears quickly and stably is set to 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. In this case, when the gazing point is the object OB (FIG. 4), the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the lines of sight RD and LD. Further, the stable focus is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. The actual field of view that the user perceives through the image display unit 20 and through the right light guide plate 26 and the left light guide plate 28 is referred to as a real field of view (FOV: Field Of View). The real field of view is narrower than the viewing angle and stable focus field, but wider than the effective field of view.

  The angle of view θ of the camera 61 of the present embodiment is set so that a wider range than the user's visual field can be imaged. The angle of view θ of the camera 61 is preferably set so that at least a range wider than the effective visual field of the user can be captured, and more preferably set so that a range wider than the actual visual field can be captured. It is more preferable that the angle of view θ of the camera 61 is set so as to be able to image a wider range than the stable viewing field of the user, and is set so as to be able to image a range wider than the viewing angle of the user's eyes Most preferred. For this reason, the camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens. The camera 61 may include a single focus lens, a zoom lens, or a lens group including a plurality of lenses.

  FIG. 5 is a block diagram functionally showing the configuration of the HMD 100. The control device 10 includes a main processor 140 that controls the HMD 100 by executing a program, a storage unit, an input / output unit, sensors, an interface, and a power supply unit 130. The storage unit, input / output unit, sensors, interface, and power supply unit 130 are connected to the main processor 140. The main processor 140 is mounted on the controller board 120 built in the control device 10.

  The storage unit includes a memory 118 and a nonvolatile storage unit 121. The memory 118 constitutes a work area for temporarily storing a computer program executed by the main processor 140 and data to be processed. The non-volatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 121 stores a computer program executed by the main processor 140 and various data processed by the main processor 140. In the present embodiment, these storage units are mounted on the controller board 120.

  The input / output unit includes a track pad 14 and an operation unit 110. The operation unit 110 includes a direction key 16, a determination key 17, and a power switch 18 provided in the control device 10. The main processor 140 controls these input / output units and acquires signals output from the input / output units.

  The sensors include a six-axis sensor 111, a magnetic sensor 113, and a GPS (Global Positioning System) receiver 115. The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Internal Measurement Unit) in which these sensors are modularized. The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor. The GPS receiver 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects coordinates of the current position of the control device 10. These sensors (six-axis sensor 111, magnetic sensor 113, GPS receiver 115) output detection values to the main processor 140 according to a sampling frequency designated in advance. The timing at which each sensor outputs a detection value may be in accordance with an instruction from the main processor 140.

  The interface includes a wireless communication unit 117, an audio codec 180, an external connector 184, an external memory interface 186, a USB (Universal Serial Bus) connector 188, a sensor hub 192, an FPGA 194, and an interface 196. ing. These function as an interface with the outside.

  The wireless communication unit 117 performs wireless communication between the HMD 100 and an external device. The wireless communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like (not shown), or is configured as a device in which these are integrated. The wireless communication unit 117 performs wireless communication complying with a standard such as a wireless LAN including Bluetooth (registered trademark) and Wi-Fi (registered trademark), for example.

  The audio codec 180 is connected to the audio interface 182 and encodes / decodes an audio signal input / output via the audio interface 182. The audio interface 182 is an interface for inputting and outputting audio signals. The audio codec 180 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. The HMD 100 according to the present embodiment outputs sound from the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 180 converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 182. The audio codec 180 converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.

  The external connector 184 is a connector for connecting an external device (for example, a personal computer, a smart phone, a game device, etc.) that communicates with the main processor 140 to the main processor 140. The external device connected to the external connector 184 can be a content supply source, and can be used for debugging a computer program executed by the main processor 140 and collecting an operation log of the HMD 100. The external connector 184 can employ various modes. As the external connector 184, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, a memory card interface, or an interface corresponding to a wireless connection such as a wireless LAN interface or a Bluetooth interface can be adopted.

  The external memory interface 186 is an interface to which a portable memory device can be connected. The external memory interface 186 includes, for example, a memory card slot in which a card-type recording medium is mounted and data is read and written, and an interface circuit. The size, shape, standard, etc. of the card type recording medium can be selected as appropriate. The USB connector 188 is an interface that can connect a memory device, a smartphone, a personal computer, or the like that conforms to the USB standard. The USB connector 188 includes, for example, a connector conforming to the USB standard and an interface circuit. The size, shape, USB standard version, etc. of the USB connector 188 can be selected as appropriate.

  Further, the HMD 100 includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor, and the like, and generates vibrations under the control of the main processor 140. The HMD 100 causes the vibrator 19 to generate a vibration with a predetermined vibration pattern when, for example, an operation on the operation unit 110 is detected or when the power of the HMD 100 is turned on / off. The vibrator 19 may be provided on the image display unit 20 side, for example, on the right holding unit 21 (the right part of the temple) of the image display unit, instead of the configuration provided in the control device 10.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted and received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196. The interface 196 is connected to each of the right display unit 22 and the left display unit 24 of the image display unit 20. In the example of the present embodiment, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is It is connected to the interface 196 of the control device 10.

  The power supply unit 130 includes a battery 132 and a power supply control circuit 134. The power supply unit 130 supplies power for operating the control device 10. The battery 132 is a rechargeable battery. The power supply control circuit 134 detects the remaining capacity of the battery 132 and controls charging of the OS 143 (FIG. 6). The power supply control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 and a detected value of the voltage of the battery 132 to the main processor 140. Note that power may be supplied from the control device 10 to the image display unit 20 based on the power supplied by the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  The right display unit 22 includes a display unit substrate 210, an OLED unit 221, a camera 61, an illuminance sensor 65, an LED indicator 67, and a temperature sensor 217. On the display unit substrate 210, an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213, and an EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 are mounted. The receiving unit 213 receives data input from the control device 10 via the interface 211. When receiving image data of an image to be displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

  The EEPROM 215 stores various data in a form that can be read by the main processor 140. The EEPROM 215 stores, for example, data on the light emission characteristics and display characteristics of the OLED units 221 and 241 of the image display unit 20, data on the sensor characteristics of the right display unit 22 or the left display unit 24, and the like. Specifically, for example, parameters relating to gamma correction of the OLED units 221 and 241 and data for compensating detection values of temperature sensors 217 and 239 described later are stored. These data are generated by the factory inspection of the HMD 100 and are written in the EEPROM 215. After shipment, the main processor 140 reads the data in the EEPROM 215 and uses it for various processes.

  The camera 61 executes imaging in accordance with a signal input via the interface 211 and outputs captured image data or a signal representing the imaging result to the control device 10. As shown in FIG. 1, the illuminance sensor 65 is provided at an end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity). As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 2). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223. The temperature sensor 217 may be built in the OLED panel 223 or the OLED drive circuit 225 (FIG. 2). For example, when the OLED panel 223 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 225, the temperature sensor 217 may be mounted on the semiconductor chip.

  The left display unit 24 includes a display unit substrate 230, an OLED unit 241, and a temperature sensor 239. On the display unit substrate 230, an interface (I / F) 231 connected to the interface 196, a receiving unit (Rx) 233, a six-axis sensor 235, and a magnetic sensor 237 are mounted. The receiving unit 233 receives data input from the control device 10 via the interface 231. When receiving image data of an image to be displayed on the OLED unit 241, the receiving unit 233 outputs the received image data to the OLED drive circuit 245 (FIG. 2).

  The 6-axis sensor 235 is a motion sensor (inertia sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized. The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor. Since the 6-axis sensor 235 and the magnetic sensor 237 are provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. The direction of the image display unit 20, that is, the field of view of the user is specified from the detected head movement.

  The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 2). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243. The temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245 (FIG. 2). Details are the same as those of the temperature sensor 217.

  The camera 61, the illuminance sensor 65, and the temperature sensor 217 of the right display unit 22, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 of the left display unit 24 are connected to the sensor hub 192 of the control device 10. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a cache function that temporarily holds the detection value of each sensor. The sensor hub 192 may be provided with a conversion function (for example, a conversion function to a unified format) of a signal format or a data format of a detection value of each sensor. The sensor hub 192 turns on or off the LED indicator 67 by starting and stopping energization of the LED indicator 67 according to the control of the main processor 140.

  FIG. 6 is a block diagram functionally showing the configuration of the control device 10. Functionally, the control device 10 includes a storage function unit 122 and a control function unit 150. The storage function unit 122 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 5). The storage function unit 122 may be configured to use the EEPROM 215 or the memory 118 in combination with the nonvolatile storage unit 121 instead of the configuration using only the storage function unit 122. The control function unit 150 is configured when the main processor 140 executes a computer program, that is, when hardware and software cooperate.

  The storage function unit 122 stores various data used for processing in the control function unit 150. Specifically, setting data 123 and content data 124 are stored in the storage function unit 122 of the present embodiment. The setting data 123 includes various setting values related to the operation of the HMD 100. For example, the setting data 123 includes parameters, determinants, arithmetic expressions, LUTs (Look Up Tables), and the like when the control function unit 150 controls the HMD 100. In addition, a use mode to be described later is stored in advance as setting data 123.

  The content data 124 includes content data (image data, video data, audio data, etc.) including images and video displayed by the image display unit 20 under the control of the control function unit 150. The content data 124 may include interactive content data. Bi-directional content refers to a user's operation acquired by the operation unit 110, processing corresponding to the acquired operation content is executed by the control function unit 150, and content corresponding to the processing content is stored in the image display unit 20. Means the type of content to display. In this case, the content data may include image data of a menu screen for acquiring a user's operation, data defining a process corresponding to an item included in the menu screen, and the like.

  The control function unit 150 executes various processes using the data stored in the storage function unit 122, whereby the OS 143, the image processing unit 145, the display control unit 147, the imaging control unit 149, and the input / output control unit 151. The functions as the average hue value calculation unit 153 and the edge tendency calculation unit 155 are executed. In the present embodiment, each functional unit other than the OS 143 is configured as a computer program executed on the OS 143.

  The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on image / video image data displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like. The image processing unit 145 may be configured by hardware different from the main processor 140 (for example, a DSP (Digital Signal Processor), in addition to a configuration realized by the main processor 140 executing a computer program.

  Note that the image processing unit 145 may execute resolution conversion processing, image adjustment processing, 2D / 3D conversion processing, and the like as necessary. The resolution conversion process is a process for converting the resolution of the image data into a resolution suitable for the right display unit 22 and the left display unit 24. The image adjustment process is a process for adjusting the brightness and saturation of image data. The 2D / 3D conversion process is a process of generating 2D image data from 3D image data or generating 3D image data from 2D image data. When these processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data, and transmits the signal to the image display unit 20 via the connection cable 40.

  The display control unit 147 generates a control signal for controlling the right display unit 22 and the left display unit 24, and controls the generation and emission of image light by each of the right display unit 22 and the left display unit 24 based on the control signal. To do. Specifically, the display control unit 147 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 147 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243 and controls the luminance of the OLED panels 223 and 243 based on the signal output from the image processing unit 145.

  The display control unit 147 controls the display mode of the pointer image in a pointer image display control process described later. In the pointer image display control process, the pointer image is displayed conspicuously so that the user can easily see the pointer image. For example, the display mode such as the color and pattern of the pointer image is controlled. A detailed description of the pointer image display control process will be described later.

  The imaging control unit 149 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores it in the storage function unit 122. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 149 acquires captured image data from the camera 61 and temporarily stores the captured image data in the storage function unit 122. In addition, the imaging control unit 149 obtains a captured image of the outside world by performing imaging of the outside world in accordance with an instruction from the display control unit 147 in a pointer image display control process described later.

  The input / output control unit 151 appropriately controls the track pad 14 (FIG. 1), the direction key 16, and the determination key 17, and acquires an input command therefrom. The acquired command is output to the OS 143 or a computer program operating on the OS 143 together with the OS 143.

  The average hue value calculation unit 153 calculates an average hue value of an area including a display position of a pointer image in a reference image described later in a pointer image display control process described later. The edge tendency calculation unit 155 calculates an edge tendency of an area including a display position of a pointer image in a reference image described later in a pointer image display control process described later.

A2. Augmented reality display:
FIG. 7 is an explanatory diagram illustrating an example of augmented reality display by the HMD 100. FIG. 7 illustrates the user's field of view VR. As described above, the image light guided to both eyes of the user of the HMD 100 forms an image on the retina of the user, so that the user visually recognizes the target image AI to be displayed as augmented reality (AR). . In the example illustrated in FIG. 7, the target image AI is an OS menu screen of the HMD 100. The menu screen includes, for example, icons for starting each application program of “message”, “phone”, “camera”, “browser”, and “navigation”. In addition, the right light guide plate 26 and the left light guide plate 28 transmit light from the outside, so that the user visually recognizes the outside world SC. As described above, the user of the HMD 100 according to the present embodiment can view the target image AI so that the portion of the visual field VR where the target image AI is displayed overlaps the external environment SC. Further, only the outside SC can be seen in the portion of the field of view VR where the target image AI is not displayed.

  As shown in FIG. 7, a pointer image Pt is displayed on the target image AI. In FIG. 7, for convenience of explanation, it is shown enlarged as compared with the pointer image Pt that is actually visually recognized by the user. The pointer image Pt is used for the user to select each menu displayed on the target image AI. In the example shown in FIG. 7, the user selects the browser menu by matching the pointer image Pt with the browser icon on the target image AI. In this state, the user can execute the browser menu by performing a tap operation or the like on the track pad 14.

  In the present embodiment, the “pointer image” is an index of an operation position related to a GUI operation and means an index indicating the operation target position in the target image AI. In the present embodiment, it means a mouse pointer image. In addition, not only a mouse pointer but an arbitrary pointer image such as a marker or a mouse cursor may be used. In the example shown in FIG. 7, the pointer image Pt has an arrow shape. The shape of the pointer image Pt is not limited to the arrow shape, and may be a geometric figure such as a circle, a triangle, or a rectangle, a symbol, or the like. Further, the pointer image Pt may have a pattern. In the present embodiment, the pattern of the pointer image Pt has directionality. Specifically, as shown in FIG. 7, the pointer image Pt has a pattern repeated in a certain direction. A detailed description of the pattern direction will be described later.

  As described above, since the user views the target image AI so as to overlap the external world SC, generally, the pointer image depends on the characteristics of the external world SC and the target image AI, for example, saturation, brightness, hue, and the like. It may be difficult to visually recognize Pt. However, in the pointer image display control process described later, the pointer image Pt is displayed by controlling the display mode of the pointer image Pt in accordance with the feature amount of the captured image of the external world SC, the target image AI, or their composite image. The visibility of the pointer image Pt is improved.

A3. Pointer image display control processing:
FIG. 8 is a flowchart showing the processing procedure of the pointer image display control processing. In the pointer image display control process, the display mode of the pointer image Pt is changed according to the use mode. The pointer image display control process is started when the user turns on the power switch 18. As illustrated in FIG. 8, the display control unit 147 acquires the usage mode from the setting data 123 (step S100).

  The usage mode is a mode for changing the display mode of the pointer image Pt. In the present embodiment, there are a total of three types of modes: a “tone change mode”, a “direction change mode”, and a “parallax change mode”. . The “color tone change mode” is a mode in which the pointer image Pt is displayed by changing the color of the pointer image Pt. The “orientation change mode” is a mode in which the direction of the pattern of the pointer image Pt is changed and the pointer image Pt is displayed. The “parallax change mode” is a mode in which the pointer image Pt is displayed by adding parallax to the pointer image Pt. The user can selectively set validity or invalidity independently for each of these three modes, and information on validity for each mode is stored in the storage function unit 122 as setting data 123. In the present embodiment, it is preset that all the three modes are valid.

  After executing step S100, the display control unit 147 determines whether or not the color tone change mode is valid (step S105). If it is determined that the color change mode is not valid (step S105: NO), the color change process (step S110) is not executed. On the other hand, when it is determined that the color change mode is valid (step S105: YES), the display control unit 147 executes a color change process (step S110).

  FIG. 9 is a flowchart showing a detailed processing procedure of the color tone changing process (step S110). As illustrated in FIG. 9, the display control unit 147 performs reference image setting processing (step S200). In the present embodiment, the “reference image” is an image in which the feature amount is considered when changing the display mode of the pointer image Pt. The reference image is set by the display control unit 147 as any one of the external image obtained by imaging, the target image AI, and the composite image of the external image and the target image AI obtained by imaging. Is done.

  FIG. 10 is a flowchart showing a detailed processing procedure of the reference image setting process (step S200). As illustrated in FIG. 10, the display control unit 147 acquires external brightness (step S300). Specifically, the display control unit 147 acquires external luminance based on a detection value corresponding to the amount of external light received by the illuminance sensor 65.

  After execution of step S300, the display control unit 147 determines whether or not the external brightness is significantly higher than the target image brightness (step S305). The target image luminance is calculated as an average value obtained by converting the RGB values of all the pixels of the target image AI into luminance values. The display control unit 147 determines a threshold value in advance, and determines that the external brightness is significantly higher than the target image brightness when the absolute value of the difference between the external brightness and the target image brightness is greater than the threshold. On the other hand, when the absolute value of the difference between the external brightness and the target image brightness is equal to or less than the threshold, it is determined that the external brightness is not significantly higher than the target image brightness. When it is determined that the external brightness is significantly higher than the target image brightness (step S305: YES), the imaging control unit 149 causes the camera 61 to capture the external environment and obtain an external image (step S310). After executing step S310, the display control unit 147 sets an external image obtained by imaging as a reference image (step S315). After execution of step S315, the reference image setting process (step S200) ends, and step S205 described later shown in FIG. 9 is executed.

  When it is determined in step S305 described above that the external brightness is not significantly higher than the target image brightness (step S305: NO), the display control unit 147 determines whether or not the external brightness is significantly lower than the target image brightness. Is determined (step S320). When it is determined that the external brightness is significantly lower than the target image brightness (step S320: YES), the display control unit 147 sets the target image AI as a reference image (step S325). After the execution of step S325, the reference image setting process (step S200) is ended in the same manner as after the above-described step S315, and step S205 described later shown in FIG. 9 is executed.

  If it is determined in step S320 described above that the external brightness is not significantly lower than the target image brightness (step S320: NO), the imaging control unit 149 captures the external environment with the camera 61 as in step S310 described above. Thus, an external image is obtained (step S330). After executing step S330, the display control unit 147 additively mixes the external image obtained by imaging and the target image AI to obtain a composite image (step S335). Note that additive color mixing is a known technique, and thus detailed description thereof is omitted. In step S335, the whole of each image is subject to additive color mixing. After executing step S335, the display control unit 147 sets the composite image as a reference image (step S340). After the execution of step S340, the reference image setting process (step S200) is terminated in the same manner as after the execution of steps S315 and S325 described above, and step S205 described later shown in FIG. 9 is executed.

  As shown in FIG. 9, after the completion of step S200, the display control unit 147 calculates an overlapping area between the reference image and the pointer image Pt (step S205). In the present embodiment, the “overlapping region” is a region including the display position of the pointer image Pt in the reference image, and a substantially circular region centering on the midpoint of the length in the longitudinal direction of the pointer image Pt. Means. In the example illustrated in FIG. 7, the area Ar1 is calculated as an overlapping area. The area Ar1 is an area that includes an area around the pointer image Pt in addition to the pointer image Pt.

  As shown in FIG. 9, after execution of step S205, the display control unit 147 acquires the hue value of the overlapping area Ar1 (step S210). Specifically, the hue value (hue angle) of each pixel in the overlapping area Ar1 is acquired. After executing step S210, the display control unit 147 calculates an average hue value of the overlapping area Ar1 (step S215). Specifically, the average hue value is calculated by dividing the sum of the hue values of the pixels in the overlapping area Ar1 acquired in step S210 described above by the number of pixels in the overlapping area.

  After executing step S215, the display control unit 147 sets the color of the pointer image Pt to the complementary color of the average hue value (step S220). In the present embodiment, “complementary color” refers to a hue whose angle (hue angle) between the hue represented by the average hue value and the hue represented by the hue value of the pointer image Pt is 180 degrees in the Munsell hue circle. Means. Further, “complementary color” means a broad concept including not only the case where the hue angle is exactly 180 degrees but also the case where the hue angle is in the range of 180 degrees ± 45 degrees. The hue ring is not limited to the Munsell hue ring, and may be, for example, a PCCS hue ring or an Ostwald hue ring. Further, the hue ring may not be continuously connected to each color such as red, yellow, green, blue, and purple.

  In step S220, when the color of the pointer image Pt is displayed as a complementary color of the average hue value, the color tone changing process (step S110) ends.

  FIG. 11 is an explanatory diagram schematically showing an example of the pointer image Pt after the color tone changing process (step S110). In FIG. 11, for convenience of explanation, the overlapping area Ar1 and the pointer image Pt in the reference image Stgd are further enlarged than the example shown in FIG. Further, the colors of the overlapping area Ar1 and the pointer image Pt are represented by hatching. As shown in FIG. 11, the pointer image Pt is displayed as a complementary color of the average hue value of the overlapping area Ar1. For this reason, the pointer image Pt is conspicuous with respect to the area | region except the pointer image Pt in the overlap area | region Ar1, and the visibility of the pointer image Pt is improving.

  As shown in FIG. 8, after executing the color tone changing process (step S110), the display control unit 147 determines whether or not the orientation change mode is valid (step S115). When it is determined that the direction change mode is not valid (step S115: NO), the direction change process (step S120) is not executed. On the other hand, when it is determined that the direction change mode is valid (step S115: YES), the display control unit 147 executes a direction change process (step S120).

  FIG. 12 is a flowchart showing a detailed processing procedure of the orientation changing process (step S120). When the direction changing process is started, first, the same processes as those in steps S200 and S205 described above are executed. Accordingly, the reference image is set and the overlapping area Ar1 is calculated.

  As shown in FIG. 12, after executing step S205, the display control unit 147 detects the edge of the overlapping area Ar1 (step S400). In step S400, edge detection is performed by using a known image processing technique such as the Canny method.

  After execution of step S400, the display control unit 147 calculates the edge tendency of the overlapping area Ar1 (step S405). In the present embodiment, “edge tendency” means a vector having an end point near the origin at the edge when the end of the reference image is the origin as a base point and an end point far from the origin as an end point. In step S405, the combined vector of all the edge trends detected in step S400 is calculated as an edge trend.

  After execution of step S405, the display control unit 147 changes the pattern of the pointer image Pt to a striped pattern that repeats in a direction intersecting with the edge tendency composite vector (step S410). After execution of step S410, the orientation change process (step S120) ends.

  FIG. 13 is an explanatory diagram schematically illustrating an example of the pointer image Pt after the orientation change process (step S120) is performed. FIG. 13 shows a case where the composite image is the reference image Stgd. In FIG. 13, the X axis indicates a direction parallel to the long side of the reference image Stgd, and the Y axis indicates a direction parallel to the short side of the reference image Stgd. In the example shown in FIG. 13, the edge tendency Edg1 of the overlapping area Ar1 in the external image is a vector parallel to the Y axis, and the edge tendency Edg2 of the overlapping area Ar1 in the target image AI is a vector parallel to the X axis. Therefore, the combined vector of the edge tendency Edg1 and the edge tendency Edg2 in the overlapping area Ar1 is a vector along the direction D1. Therefore, in step S410, the pointer image Pt is displayed in a striped pattern in which the pattern of the pointer image Pt is repeated in the direction D2 perpendicular to the direction D1.

  As illustrated in FIG. 8, after executing the orientation change process (step S120), the display control unit 147 determines whether or not the parallax change mode is valid (step S125). When it is determined that the parallax change mode is not valid (step S125: NO), the parallax change process (step S130) is not executed, and the process returns to the above-described step S100. On the other hand, when it is determined that the parallax change mode is valid (step S125: YES), the display control unit 147 executes a parallax change process (step S130).

  FIG. 14 is a flowchart showing a detailed processing procedure of the parallax change processing (step S130). As shown in FIG. 14, when the parallax change process is started, first, the same process as in step S200 described above is executed. Therefore, a reference image is set. After executing step S200, the display control unit 147 determines whether or not the above-described color tone changing process (step S110) has been executed (step S500). When it is determined that the color tone changing process is not executed (step S500: NO), the display control unit 147 calculates the control target coordinates in the reference image and the hue value of the surrounding area (step S505). In the present embodiment, “control target coordinates” mean coordinates that are targets of parallax change processing in the pointer image Pt. Further, the “peripheral area” means an area including an area where the pointer image Pt is expected to move when the pointer image Pt is displayed with parallax. In the present embodiment, the maximum movement area of the pointer image Pt when displaying the pointer image Pt with parallax is set in advance as the peripheral area. In step S505, the display control unit 147 calculates the control target coordinates in the reference image and the hue value of the surrounding area in the same procedure as in step S210 in the above-described color tone changing process (step S110).

  After execution of step S505, the display control unit 147 calculates an average hue value of the control target and the surrounding area (step S510). In step S510, similar to step S215 in the above-described color tone changing process (step S110), the sum of the hue values calculated in step S505 is taken, and the average hue value is divided by the number of pixels in the control target and the surrounding area. calculate.

  After execution of step S510, the display control unit 147 sets the average hue value of the control target and the surrounding area to a complementary color of the average hue value (step S515). In step S515, similar to step S220 in the above-described color tone change mode (step S100), a complementary color of the average hue value of the control target and its surrounding area is set.

  After executing step S515, the display control unit 147 displays the pointer image Pt as a solid with parallax (step S520). Specifically, the display control unit 147 displays the pointer image Pt as a three-dimensional image with a parallax based on the default value of the parallax change amount preset in the display control unit 147.

  After execution of step S520, the display control unit 147 moves the focal position of the pointer image Pt displayed as a three-dimensional object by a predetermined distance in the direction from the image display unit 20 to the outside world, and from the outside world to the image display unit 20. Are moved alternately by a predetermined distance in the direction toward (step S525). Specifically, the focal position between the pointer image Pt and the user when the pointer image Pt is displayed on the image display unit 20 is the reference position, and the focal position is a predetermined distance in the direction from the reference position toward the outside world. Move. Further, the moved focal position is moved by a predetermined distance in the direction from the outside to the image display unit 20. After execution of step S525, the parallax change process (step S130) ends, and the process returns to step S100 described above as shown in FIG.

  When it is determined in step S500 described above that the color tone changing process has been executed (step S500: YES), the display control unit 147 executes steps S520 and S525 described above, and the parallax changing process (step S130) ends. Then, as shown in FIG. 8, the process returns to step S100 described above.

  Note that the above-described steps S505 to S515 are processes performed to prevent the control target coordinates from becoming difficult to understand by executing the above-described steps S520 and S525. Further, these processes correspond to steps S210 to S220 in the color tone changing process (step S110) shown in FIG. For this reason, as shown in FIG. 14, when the color tone changing process (step S110) has already been executed, these processes are not executed again in the parallax changing process (step S130).

  FIG. 15 is an explanatory diagram schematically illustrating an example of the pointer image Pt after execution of the parallax changing process (step S130). In FIG. 15, the pointer image Pt shows a state where the focal position is not moved. The pointer image Pt1 moves the focal position by a predetermined distance in the direction from the image display unit 20 to the outside world, and the pointer image Pt2 moves the focal position by a predetermined distance in the direction from the outside world to the image display unit 20. Each state is shown.

  As shown in FIG. 15, the focal length between the pointer image Pt and the right eye RE and the left eye LE of the user is the focal length w. The focal length w1 between the pointer image Pt1 and the user's right eye RE and left eye LE is larger than the focal length w. For this reason, since the pointer image Pt1 appears farther from the user than the pointer image Pt, the pointer image Pt1 can be viewed in a shape in which the pointer image Pt is recessed. On the other hand, the focal length w2 between the pointer image Pt2 and the user's right eye RE and left eye LE is smaller than the focal length w. For this reason, since the pointer image Pt2 appears closer to the user than the pointer image Pt, the pointer image Pt2 can be visually recognized in a shape that projects the pointer image Pt.

  According to the HMD 100 of the first embodiment described above, any one of the external image obtained by imaging, the target image AI, and the composite image of the external image and the target image AI is used as the reference image Stgd. The display mode of the pointer image Pt is changed and displayed according to the feature amount (average hue value and edge tendency) of the area Ar1 including the display position of the pointer image Pt in the reference image Stgd. For this reason, the pointer image Pt can be displayed by the display mode in consideration of the characteristics of the reference image Stgd, and the visibility of the pointer image Pt can be improved.

  Further, since the reference image Stgd is a composite image, the display mode of the pointer image Pt can be controlled in consideration of the feature amounts of the external image and the target image AI. Therefore, the visibility of the pointer image Pt can be improved when the target image AI is displayed in a superimposed manner on the external world that is visible through the screen and the pointer image Pt is displayed in a superimposed manner.

  In addition, since the pointer image Pt is displayed with the color of the pointer image Pt complementary to the average hue value, the pointer image Pt can be more easily seen as compared with a configuration in which the color of the pointer image Pt is not complementary to the average hue value. .

  Further, since the pointer image Pt is displayed with the pattern of the pointer image Pt having a direction determined based on the edge tendency, the visibility of the pattern of the pointer image Pt can be improved and the visibility of the pointer image Pt itself can be improved. .

  In addition, since the pointer image Pt is displayed by making the pattern of the pointer image Pt a striped pattern that is repeated in a direction perpendicular to the combined vector of edge trends, the visibility of the striped pattern of the pointer image Pt is improved and the pointer image Pt itself is displayed. Visibility can be improved.

  In addition, the focal position of the pointer image Pt displayed as a three-dimensional object is moved by a predetermined distance in a direction from the image display unit 20 toward the outside world, and moved by a predetermined distance in a direction from the outside world to the image display unit 20. Since the pointer image Pt displayed as a three-dimensional object is alternately displayed, the pointer image Pt can be displayed alternately so that the pointer image Pt can be displayed in a protruding manner.

B. Second embodiment:
B1. Overall configuration of the transmissive display device:
Since the head-mounted display device 100 in the second embodiment is the same as the head-mounted display device 100 in the first embodiment shown in FIG. 1, detailed description thereof is omitted.

B2. Pointer image display control processing:
FIG. 16 is a flowchart showing a detailed processing procedure of the parallax change processing (step S130a) in the second embodiment. The pointer image display control process in the second embodiment is different from the parallax change process (step S130) in the first embodiment in the procedure of the parallax change process, and other process procedures are the same as those in the first embodiment. The parallax change process (step S130a) of the second embodiment is the same as the parallax change process (step S130) of the first embodiment in that step S516 and step S517 are added and executed, and step S525 is omitted. Different. Other procedures in the parallax changing process (step S130a) of the second embodiment are the same as those of the parallax changing process (step S130) of the first embodiment, and therefore the same procedures are denoted by the same reference numerals and the details thereof are described. The detailed explanation is omitted.

  In the parallax changing process of the second embodiment, the pointer image Pt is displayed by alternately switching between displaying the pointer image Pt as a solid with parallax and displaying it as a surface without parallax. Specifically, as shown in FIG. 16, when the control target coordinates and the surrounding area are made complementary colors of the average tone value (step S515), the display control unit 147 changes the amount of parallax based on the average hue value. Is set (step S516). Specifically, the display control unit 147 calculates a difference between the average hue value of the pointer image Pt and the average hue value of the overlapping area Ar1, and when the calculated difference in average hue value is larger than a predetermined threshold value, The amount of change in parallax is set to a relatively small fixed value. When the calculated difference in average hue value is equal to or less than a predetermined threshold, the parallax change amount is set to a relatively large fixed value. When the difference between the average hue value of the pointer image Pt and the average hue value of the overlapping area Ar1 is large, the pointer image Pt can be easily identified even if the amount of change in parallax is small, whereas the average hue value of the pointer image Pt When the difference from the average hue value of the overlapping area Ar1 is small, the pointer image Pt may not be identified unless the amount of change in parallax is increased. Therefore, as described above, when the calculated average hue value difference is large, the parallax change amount is set to a relatively small value, and when the calculated average hue value difference is small, the parallax change amount is relatively set. Set to a large value.

  After executing step S516, the display control unit 147 sets a period for applying parallax based on the average hue value (step S517). Specifically, as in step S516 described above, the display control unit 147 calculates the difference between the average hue value of the pointer image Pt and the average hue value of the overlapping area Ar1, and the difference between the calculated average hue values is calculated. If it is larger than the predetermined threshold, the parallax cycle is set to be relatively short. When the calculated difference in average hue value is equal to or less than a predetermined threshold, the parallax cycle is set to be relatively long.

  Note that the parallax change amount and the parallax cycle are preferably set based on 3DC safety guidelines (ISO ISO 3 compliant) formulated by the 3D Consortium. Specifically, the parallax is preferably set in a range equal to or less than the inter-pupil distance, and more preferably set in a range corresponding to 5 cm. The change amount of the parallax angle is preferably set within 2 degrees, and more preferably set within 1 degree which is a comfortable parallax range.

  After execution of step S517, the pointer image Pt is displayed as a solid with parallax, as in step S520 described above. At this time, the pointer image Pt is displayed based on the change amount and period of the parallax set in steps S516 and S517 described above.

  According to the HMD 100 of the second embodiment described above, the pointer image Pt is displayed as a solid with parallax, and the amount of change in parallax is set based on the average hue value. When Pt moves, the visibility of the pointer image Pt can be improved, and the followability to the pointer image (performance to follow with the eyes) can be improved.

  In addition, the display of the pointer image Pt as a solid with parallax and the display of the pointer image Pt as a non-parallax surface are alternately switched, and a period for applying parallax is set based on the average hue value Therefore, the visibility can be improved by making the pointer image Pt stand out. Further, even when the pointer image Pt moves to a region having a different average hue value, the period for applying the parallax is changed, so that the pointer image Pt can be made conspicuous, and the followability to the pointer image Pt can be improved.

C. Third embodiment:
C1. Overall configuration of the transmissive display device:
Since the head-mounted display device 100 in the third embodiment is the same as the head-mounted display device 100 in the first embodiment shown in FIG. 1, detailed description thereof is omitted.

C2. Pointer image display control processing:
FIG. 17 is a flowchart showing a detailed processing procedure of the parallax change processing (step S130b) in the third embodiment. The pointer image display control process in the third embodiment is different from the parallax change process (step S130) in the first embodiment in the procedure of the parallax change process, and other process procedures are the same as those in the first embodiment. The parallax change process (step S130b) of the third embodiment is different from the parallax change process (step S130) of the first embodiment in that step S525a is executed instead of step S525. Other procedures in the parallax changing process (step S130b) of the third embodiment are the same as those of the parallax changing process (step S130) of the first embodiment. The detailed explanation is omitted.

  In the parallax changing process of the third embodiment, the brightness of the tip portion of the pointer image Pt is periodically changed and displayed. Specifically, as shown in FIG. 17, when the pointer image Pt is displayed as a solid with parallax (step S520), the display control unit 147 periodically changes the brightness of the tip portion of the pointer image Pt. Change (step S525a). More specifically, the display control unit 147 changes the luminance of the tip portion of the pointer image Pt within a predetermined range. As a result, the tip of the pointer image Pt blinks and is displayed to the user.

  According to the HMD 100 of the third embodiment described above, since the brightness of at least a part of the pointer image Pt is periodically changed and displayed, a part of the pointer image Pt can be blinked and displayed. The visibility of the pointer image Pt can be further improved.

D. Variation:
D1. Modification 1:
In each of the above embodiments, the reference image is a composite image, but the present invention is not limited to this. For example, the reference image may be the target image AI or an external image obtained by imaging. For example, in the above-described reference image setting process (step S200), when the external brightness is compared with the target image brightness (step S305, step S320), if the external brightness is lower than a predetermined threshold, the color and brightness of the external environment In this case, the reference image may be used as the target image AI. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D2. Modification 2:
In the direction changing process (step S120) in each of the above embodiments, the direction in which the striped pattern of the pointer image Pt is repeated is a direction perpendicular to the composite vector of the edge tendency, but may not be a perpendicular direction. For example, the direction vector may be a direction that intersects the combined vector of the edge tendency at an angle different from 90 degrees. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D3. Modification 3:
In the first embodiment, the pointer image Pt is displayed in a striped pattern, but the present invention is not limited to this. For example, the pattern of the pointer image Pt may be displayed in a gradation.

  FIG. 18 is an explanatory diagram schematically showing the pattern of the pointer image Pt3 in the third modification. The pointer image Pt3 shown in FIG. 18 differs from the pointer image Pt of the first embodiment shown in FIG. 13 only in the pattern, and the other components in FIG. 18 are the same as the other components in FIG. As shown in FIG. 18, the pointer image Pt3 is directed from the front end s1 to the rear end e1 of the pointer image Pt3, that is, along the direction D2 perpendicular to the direction D1 of the edge-trending composite vector. The color density is displayed so as to become gradually lighter. As described above, the pointer image Pt3 may be displayed so as to have different colors (densitys) in layers along the direction determined based on the edge tendency, that is, the direction D2 perpendicular to the edge tendency composite vector direction D1. . Even in such a configuration, the same effects as those of the first embodiment can be obtained.

D4. Modification 4:
In the third embodiment, the brightness of the tip portion of the pointer image Pt is periodically changed and displayed. However, the brightness is periodically changed and displayed only at the tip portion of the pointer image Pt. I can't. For example, the entire pointer image Pt may be used. That is, in general, the same effects as those of the third embodiment can be obtained as long as the brightness of at least a part of the pointer image Pt is periodically changed.

D5. Modification 5:
In each of the above embodiments, the overlapping area Ar1 is an area including the display position of the pointer image Pt in the reference image, and is a substantially circular area centering on the midpoint of the length in the longitudinal direction of the pointer image Pt. However, the present invention is not limited to this. For example, it is not limited to a substantially perfect circle shape, but may be any other shape such as an elliptical shape or a polygonal shape. Further, for example, it may be a region where the outer edge is a set of points where the outer edge of the pointer image Pt is separated by a predetermined distance. Further, for example, an area that does not include the peripheral area of the pointer image Pt may be used. That is, an area formed along the outline of the pointer image Pt may be an overlapping area. For example, when the pointer image Pt overlaps an image such as an icon, an area including the icon or the like may be set as an overlapping area. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D6. Modification 6:
In each of the above embodiments, the pointer image display control process is started when the power of the HMD 100 is turned on, but the present invention is not limited to this. For example, the pointer image display control process may be started upon detection of a predetermined operation in the HMD 100. As the predetermined operation, for example, a tap operation on the track pad 14 in the control device 10 or activation of a predetermined application is applicable. Even in such a configuration, the same effects as those of the above embodiments can be obtained. In addition, since the control of the display mode of the pointer image Pt is executed in response to detection of a predetermined operation in the HMD 100, the user of the HMD 100 can adjust the trigger of executing the control of the display mode of the pointer image Pt. Convenience can be improved. Furthermore, since the visibility of the pointer image Pt can be improved by performing a predetermined operation only when the user has difficulty in visually recognizing the pointer image Pt, the processing of the HMD 100 can be performed compared to the configuration in which the pointer image display control processing is always performed. The load can be reduced.

D7. Modification 7:
In each of the above embodiments, three use modes of the pointer image display control process are the color tone change mode, the direction change mode, and the parallax change mode, but the present invention is not limited to this. For example, the color change mode may be omitted and the orientation change mode and the parallax change mode may be used. For example, only the color tone change mode may be used. Even in such a configuration, since the display mode of the pointer image Pt is changed and displayed according to the feature amount of the region including the display position of the pointer image Pt in the reference image, the same effects as those in the above embodiments are obtained. .

D8. Modification 8:
In each of the above embodiments, the composite image is a composite image of the entire external image and the entire target image AI, but the present invention is not limited to this. For example, it may be a composite image of an area including the display position of the pointer image Pt in the external image and an area including the display position of the pointer image Pt in the target image AI. Even in such a configuration, the same effects as those of the above embodiments can be obtained. In addition, the processing area can be narrowed, and the processing load of the pointer image display control process can be reduced.

D9. Modification 9:
In each of the above embodiments, the composite image is generated by additive color mixture, but the present invention is not limited to this. For example, it may be generated by additive color reduction. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D10. Modification 10:
In each of the above embodiments, the reference image setting process (step S200) is executed in each of the color tone changing process, the direction changing process, and the parallax changing process, but the present invention is not limited to this. For example, when a plurality of modes are used as the use mode, the reference image setting process is executed in the display mode changing process executed first, and the reference image setting executed first in the other display mode changing process executed thereafter The processing result of the processing may be used. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D11. Modification 11:
In each of the above embodiments, the display device that executes the pointer image display control process is the HMD 100, but the present invention is not limited to this. For example, a head-up display (HUD) or a video see-through HMD may be used. Further, a stationary transmission type display device may be used. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D12. Modification 12:
In each of the above embodiments, at least part of the function of the display control unit 147 may be executed by another control function unit. Specifically, in each of the embodiments described above, the display control unit 147 executes the display of images by the OLED panels 223 and 243 and the pointer image display control process. For example, the pointer image display control process May be executed by another control function unit. Moreover, you may implement | achieve some or all of the function and process of these control function parts using digital circuits, such as CPU, ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array). Even in such a configuration, the same effects as those of the above embodiments can be obtained.

D13. Modification 13:
In the second embodiment, the amount of change in the parallax and the period of the parallax are less than the predetermined threshold when the difference between the average hue value of the pointer image Pt and the average hue value of the overlapping area Ar1 is larger than a predetermined threshold. Although the predetermined value is set for each case, the present invention is not limited to this. For example, the amount of change in parallax and the period of parallax may be set to decrease as the difference in average hue value increases, or set to increase as the difference in average hue value decreases. Also good. For example, it may be set so as to be proportional to the difference in average hue value. Even in such a configuration, the same effects as those of the second embodiment can be obtained.

D14. Modification 14:
In the color tone changing process, the direction changing process, and the parallax changing process in each of the above embodiments, the display process of the pointer image Pt after the change of the display mode is performed, but the present invention is not limited to this. For example, in each of the color tone changing process, the orientation changing process, and the parallax changing process, the display mode of the pointer image Pt may be changed, and then the changed pointer image Pt may not be displayed. In this case, you may display the pointer image Pt after execution of the display mode change process performed last among each display mode change process. Specifically, pointer image Pt display processing may be performed after execution of step S130 shown in FIG. Even with such a configuration, the same effects as those of the above embodiments can be obtained.

D15. Modification 15:
In the orientation changing process in each of the above embodiments and modifications, the pattern of the pointer image Pt is changed, but the present invention is not limited to this. For example, the shape of the pointer image Pt may be changed to a shape based on the direction intersecting with the edge tendency composite vector. For example, when the shape of the pointer image Pt is a substantially perfect circle shape, the shape of the pointer image Pt may be changed to an elliptical shape that is elongated along the direction intersecting the combined vector of the edge tendency. For example, both the pattern and shape of the pointer image Pt may be changed, or only the shape of the pointer image Pt may be changed. Even in such a configuration, the same effects as those of the above-described embodiments and modifications can be obtained.

D16. Modification 16:
In each of the above embodiments, the display of the pointer image Pt with parallax is periodically repeated, but the present invention is not limited to this. For example, the pointer image Pt with parallax may be displayed only once. Further, for example, after displaying the pointer image Pt with the parallax once, the display of the pointer image Pt with the parallax may be continued without returning the display mode of the pointer image Pt. Even in such a configuration, the same effects as those of the above embodiments can be obtained.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

    DESCRIPTION OF SYMBOLS 10 ... Control apparatus, 12 ... Lighting part, 14 ... Trackpad, 16 ... Direction key, 17 ... Decision key, 18 ... Power switch, 19 ... Vibrator, 20 ... Image display part, 21 ... Right holding part, 22 ... Right display Unit: 23 ... Left holding unit, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... Headset, 32 ... Right earphone, 34 ... Left earphone, 40 ... Connection Cable, 46 ... Connector, 61 ... Camera, 63 ... Microphone, 65 ... Illuminance sensor, 67 ... LED indicator, 100 ... Head-mounted display device, 110 ... Operation unit, 111 ... 6-axis sensor, 113 ... Magnetic sensor, 115 ... GPS receiver, 117 ... wireless communication unit, 118 ... memory, 120 ... controller board, 121 ... non-volatile storage unit, 122 ... storage function unit, 23 ... Setting data, 124 ... Content data, 130 ... Power supply unit, 132 ... Battery, 134 ... Power supply control circuit, 140 ... Main processor, 145 ... Image processing unit, 147 ... Display control unit, 149 ... Imaging control unit, 150 ... Control function unit, 151 ... I / O control unit, 153 ... Average hue value calculation unit, 155 ... Edge tendency calculation unit, 180 ... Audio codec, 182 ... Audio interface, 184 ... External connector, 186 ... External memory interface, 188 ... USB Connector, 192 ... Sensor hub, 196 ... Interface, 210 ... Display unit board, 211 ... Interface, 213 ... Receiver, 215 ... EEPROM, 217 ... Temperature sensor, 221 ... OLED unit, 223 ... OLED panel, 225 ... OLED drive circuit 2 DESCRIPTION OF SYMBOLS 0 ... Display unit board | substrate, 231 ... Interface, 233 ... Reception part, 235 ... 6 axis sensor, 237 ... Magnetic sensor, 239 ... Temperature sensor, 241 ... OLED unit, 243 ... OLED panel, 245 ... OLED drive circuit, 251 ... Right Optical system, 252 ... Left optical system, 261 ... Half mirror, 281 ... Half mirror, AI ... Target image, Ar1 ... Overlapping region, D1 ... Direction, D2 ... Direction, EL ... End, ER ... End, Edg1 ... Edge Trend, Edg2 ... Edge tendency, L ... Image light, LD ... Line of sight, LE ... Left eye, OB ... Object, OL ... External light, Pt ... Pointer image, Pt1 ... Pointer image, Pt2 ... Pointer image, Pt3 ... Pointer image RD: line of sight, RE: right eye, SC: outside, Stgd: reference image, VR: field of view, e1: rear end, s1: tip, w ... focal length, w1 ... focal length, w2 ... focal length

Claims (12)

A transmissive display device,
A display unit that has light transparency and displays a target image and a pointer image that are to be displayed in a superimposed manner on an external environment that is visible through the self,
A display control unit for controlling the display mode of the pointer image;
An imaging unit for imaging the outside world;
With
The display control unit uses the pointer in the reference image as any one of the external image obtained by the imaging, the target image, and the composite image of the external image and the target image. In accordance with the feature amount of the area including the display position of the image, the display mode of the pointer image is changed and displayed.
A transmissive display device. The transmissive display device according to claim 1,
The reference image is the composite image.
A transmissive display device. The transmissive display device according to claim 1 or 2,
An average hue value calculating unit that calculates an average hue value of an area including the display position of the pointer image in the reference image;
The display control unit displays the pointer image with the color of the pointer image as a complementary color of the average hue value;
A transmissive display device. The transmissive display device according to any one of claims 1 to 3,
An edge tendency calculating unit that calculates an edge tendency of an area including the display position of the pointer image in the reference image;
The display control unit displays the pointer image in a pattern having a directionality determined based on the edge tendency as a pattern of the pointer image;
A transmissive display device. The transmissive display device according to claim 4,
The display control unit displays the pointer image in a striped pattern that is repeated in a direction intersecting the combined vector of the edge tendency.
A transmissive display device. In the transmissive display device according to any one of claims 1 to 5,
An average hue value calculating unit that calculates an average hue value of an area including the display position of the pointer image in the reference image;
The display control unit displays the pointer image as a solid with a parallax, and sets the amount of change in the parallax based on the average hue value.
A transmissive display device. The transmissive display device according to any one of claims 1 to 6,
An average hue value calculating unit that calculates an average hue value of an area including the display position of the pointer image in the reference image;
The display control unit alternately switches between displaying the pointer image as a solid with parallax and displaying the pointer image as a non-parallax surface, and based on the average hue value, the parallax Set the cycle
A transmissive display device. The transmissive display device according to any one of claims 1 to 7,
The display control unit
Display the pointer image as a solid with parallax,
Moving the focal position of the pointer image displayed as the three-dimensional object by a predetermined distance in a direction from the display unit toward the outside world, and moving the predetermined position in a direction from the outside world to the display unit; , Alternately,
A transmissive display device. The transmissive display device according to any one of claims 1 to 8,
The display control unit periodically changes the brightness of at least a part of the pointer image to display the pointer image;
A transmissive display device. The transmissive display device according to any one of claims 1 to 9,
The display control unit executes control of a display mode of the pointer image triggered by detection of a predetermined operation in the transmissive display device.
A transmissive display device. A display control method in a transmissive display device having a light transmission property and including a display unit that displays a target image to be displayed and a pointer image so as to be superimposed on an external environment that is visible through the device.
Imaging the outside world to obtain an outside world image;
Using one of the external image, the target image, and the composite image of the external image and the target image as a reference image, the feature amount of the region including the display position of the pointer image in the reference image In response, the step of changing the display mode of the pointer image to display,
Comprising
Display control method. Computer for realizing a display control method in a transmissive display device having a light transmitting property and including a display unit that displays a target image to be displayed and a pointer image so as to be superimposed on an external environment that is visible through the image. A program,
A function to capture the outside world and obtain an outside world image;
Using one of the external image, the target image, and the composite image of the external image and the target image as a reference image, the feature amount of the region including the display position of the pointer image in the reference image In response, a function to change the display mode of the pointer image and display on the display unit;
To make the computer
Computer program.

Images