NL2034292B1

NL2034292B1 - See-through device and method for enhanced visual perception

Info

Publication number: NL2034292B1
Application number: NL2034292A
Authority: NL
Inventors: Max Pieter Hoogsteen Karst
Original assignee: Angstone B V
Priority date: 2023-03-08
Filing date: 2023-03-08
Publication date: 2024-09-20
Also published as: WO2024186199A1

Abstract

The invention relates to a see-through device (1) for enhanced visual perception that comprises an input module (2) that can capture an image of the real-world, a processing module (3) that can analyze and modify optical properties of the image to create a modified image, and a see-through output module (4) that comprises a combiner unit (38) and a display engine (40) that can overlay a view of the real- world with the modified image to create an overlayed view. According to the invention the processing module (2) can analyze which areas in the image are near or outside the limit of the dynamic range of the human eye and can modify those areas, so that they are redistributed within the dynamic range of the human eye and where the output module (4) can selectively overlay the view of the real-world with these areas of the modified image.

Description

See-through device and method for enhanced visual perception

Description

The invention relates to a see-through device for enhanced visual perception that comprises an input module that can capture an image of the real-world, a processing module that can analyze and modify the optical properties of the image to create a modified image, and a see-through output module that comprises a combiner unit and a display engine that can overlay the view of the real-world with the modified image to create an overlayed view,. The invention also relates to a method to enhance visual perception.

Such a device is known from WO20198017861A1, where a wearable display is described with a mounted optical sensor system that captures an image of the real world, modulates the image, and then displays that image in real time into the user's view of the real-world. The image modulation enables image data of objects at multiple focal distances from the user to be reconstructed into an image with objects at a focal distance that the user can perceive and differentiate. The known device has a video processing unit with algorithms designed to modulate still and moving video images so that these can be used by persons with genetic, physiological, and psychological conditions that result in visual deficits.

The known device has the problem that the overlay between the modulated image and the view of the real-world is misaligned, leading to double vision or distortion that results in decreased visual perception and spatial awareness.

According to the present invention the device is characterized in that the processing module can analyze which areas in the image are near or outside the limit of the dynamic range of the human eye and can modify those areas, so that they are redistributed within the dynamic range of the human eye and where the output module can selectively overlay the view of the real-world with these areas of the modified image. The real-world view can be overlayed by areas of the modified image on a pixel by pixel basis. Thus, in the inventive device the modified image only comprises those areas that before modification are near or outside the limit of the dynamic range of the human eye. Modifying the areas near or outside the limit of the dynamic range of a human eye so they are redistributed inside that range, makes contrast differences in those areas visible and identifiable to the human eye. The user thus sees an overlayed view that is a combination of the real-world view overlayed with modified areas where contrast differences were not visible before modification.

Based on user and scene specific threshold values an algorithm will check all intensity values of the captured image to determine which pixels are near or outside the limit of the perceivable dynamic range of a user. These thresholds are user and scene specific because the perceivable dynamic range can vary from person to person based on age, other factors like the presence of visual impairments, the light adaptation state of the eye and the lighting conditions of the scene.

Here near the limit of the dynamic range of the human eye means the outer 25% of luminance levels, on either end of the range, where contrast differences can be perceived, although with some difficulty, in any one view at a given moment in time, i.e., without adaptation by the eye by processes such as the modulation of the pupil size which can take up to 30 minutes to complete. Here outside the dynamic range means luminance levels beyond either end of the range, where contrast differences cannot be perceived in any one view at a given moment in time. Without adaptation the eye is generally able to detect contrast differences of 1000 to 1. Here 1000 is the most luminous area and 1 the least luminous. Please note that this range can vary from person to person based on age and presence of any visual impairments. Contrast differences outside of that range are difficult or impossible to perceive in any one view ata given moment in time. They are perceived as black (low luminance) or white (high luminance), thereby, losing the contrast differences and any visual cues that may be discerned in those areas. However, many real-world scenes feature a larger dynamic range, for example contrast differences of 1000000 to 1.

The device can be used with a pre-set of values for the perceivable dynamic range of the human eye. It is advantageous if the device comprises a control unit that can adjust the properties of modules to accommodate for different lighting conditions in the real- world and for different sensitivities of an eye of a user. The sensitivity of the human eye is depending on the lighting conditions, for instance the size of the pupil can change to modulate the amount of light on the retina, thereby shifting the dynamic range. Further the eyes of the user can have specific properties, i.e. different visual function, such as extra or low sensitivity for light, like photophobia or night blindness.

These properties can result from aging or the presence of visual impairments. The threshold values of the dynamic range of a user's eye can for instance be manually set by the user by means of a calibration mode which should be activated when observing a particular scene through the device over a period of time. The user can save these values so the device can automatically apply these thresholds as soon as a scene with similar lighting conditions is detected. The lighting conditions of a scene can be detected by the processing module because the processing module is at all times aware of the settings of the input module, such as shutter speed, and pre- programmed to relate the resulting intensity values of the image to absolute luminance values of a scene. The device can also have presets of values that can be selected based on information provided by the user such as age, presence of visual impairments or results from visual function testing by their health care provider.

Preferably the device has a camera aimed at the eye of a user to predict the light adaptation state of the eye by matching the size of the pupil to preset user specific values. Such user specific values can for instance be set in the calibration mode of the device in a particular scene. Based on the size of a user’s pupil the device can predict the perceivable dynamic range for a particular scene and lighting conditions. In embodiments without this additional sensor when the device is started it will set the brightness of the display to a low value, and if required, gradually increase it towards the required level for a particular scene. This is done not to risk damaging or irritating the eye. A user may manually overwrite this through the control unit in case there is a mismatch between the preferable brightness of the display and the light adaptation state of the eye of a user.

The redistribution of the dynamic range is based on the intensity values of the image after modification by the processing module and executed through the overlay of the modified image by the display engine and combiner unit. Please note that the specification of the input module and output module such as the number of bits per pixel value and the contrast ratio can affect this redistribution.

It was found that an overlay with the complete image as in the prior art, i.e. with all areas regardless of the dynamic range, interferes with the user’s real see through view and causes problems.

Misaligned image areas cause double vision or distortion. This prevents the user to properly see and identify areas or objects in the field of view. In the inventive device only areas of which the contrast cannot be easily perceived by the user, are modified and overlayed onto the view of the real-world. Here, not easily perceived refers to the afore mentioned outer 25% on either side of the dynamic range of the eye. Since only those areas that are not easily perceived are shown the perceivable misalignment in the overlayed view between the modified image and the real world view is hon-existent or minimized. This prevents double vision and distortion as the contrast differences of the modified areas are not easily visible to the eye in the real-world view. In the prior art, always the whole modified image is added to the real world view, so also areas that are already visible are overlayed with a modified image causing distortion and double vision.

The known device will also result in decreased visual perception when displaying images when bright objects are within the field of view. The whole image including specific objects will then appear so bright that no details can be discerned. Compare this to looking into bright sunshine, when all other objects will disappear. Also, details perceivable without the device maybe obscured or occluded by the overlayed view,

Since the dynamic range of cameras is also limited, details beyond this range will be washed out and lost. Overlaying such washed out areas on top of the real world view can obscure details that are already perceivable without the device, thereby, decreasing the perceivable dynamic range. In the device of the present invention such bright areas can be filtered out of the image so they are not overlayed on top of the real world view thereby preventing these problems. Additionally, bright areas can be overlayed with the same area but with less brightness, so that it falls within the dynamic range the eye can perceive.

Preferably the processing module can adapt the modified image to correct for the difference between specifications of the input and output modules and a parallax resulting from the difference between the optical center of the input module and the optical center of the eye of a user. Such a correction can be based on a (controllable) distance between the user and the areas observed. Thereby further minimizing and eliminating any remaining misalignment between the modified image and the real- world view. Here, adapt the modified image means that the processing module can perform a 2D or 3D transformation, which can include scaling, cropping, shifting and rotating the modified image as to get a good aligned overlay between the modified image and the real-world view. The adaptation will normally be done by using the control unit via an input unit for the control unit. Such an input unit can be physically present on the device or can be used wirelessly via for instance a special app for the device. Less likely, but also possible is it to align the modified image and the view of the real-world by using optical elements such as lenses, (semi-transparent) mirrors and prisms. For example, placing a semi-transparent mirror at an angle in front of the eye through which the user can see the light of the real-world, while at the same time part of that light from the real-world is reflected into the camera of the input module.

Further adaptations could be applied through lens magnification and shifting or rotating 5 using a prism.

Due to the architecture of the device, input and output modules cannot be aligned physically without obstructing the real-world view as the optical center of the input module should not be positioned within the line of sight, right in front of the user's eye.

Thus, there is a parallax that results from the difference between the optical center of the input module that captures the image and the optical center of the real-world view through the device. This parallax needs to be corrected.

According to a further embodiment of the invention the modified image contains a marker that can be used to facilitate adaptation of the modified image. The marker makes it very easy to adapt the modified image to correct for the difference between specifications of the input and output modules and a parallax resulting from the difference between the optical center of the input module and the optical center of the eye of a user.

Preferably the outer surface of the combiner unit of the output module comprises an optical filter that filters light from the real-world view. The outer surface of the combiner is that surface that faces the real-world. When the device is being used as a device adapted for specific circumstances, like used as sunglasses, the device can make contrast differences in those areas that would normally not be seen, or difficult to perceive, visible to the user. Thus, very bright areas and areas in the shadows, where contrast differences would not be easily perceivable through normal sunglasses can be seen clearly with the inventive device. From the outside it will look like the user is wearing normal sun glasses, but he/she will have the added benefit that he/she can see clearly.

In an especially advantageous embodiment, the optical filter comprises an adaptable optical filter. Such an adaptable filter can be for instance a photochromic, smart film or display filter. The adaptable optical filter can change transmission properties depending on external triggers. Photochromic filters change their light transmission depending on the light intensity falling on them. The light transmission of smart or polymer dispersed liquid crystal (PDLC) films can be adjusted by applying a voltage across the film. Displays like e-ink can also be used to make more or less transmission through the filter possible in a controlled way. Such a use of adaptable optical filters can make the use of the device much broader, further increasing the dynamic range that can be perceived. With little light transmission the device can be used in bright sunlight, whereas with full light transmission the device can be used in low light circumstances. All the while keeping the advantage that all over the image the user can clearly see all areas with a dynamic range perceivable by the eyes.

In a preferred embodiment the optical filter comprises a film or display with semi- transparent regions that can be individually darkened. Here transmission of individual areas can be modified to filter out bright lights. This is especially useful in a device used for instance for nightly driving a car. Light from oncoming vehicles can be very bright, especially vehicles with LED headlights. The processing unit can then give signals to those regions to darken where the bright lights are visible to the user. Thus preventing that a bright spot would temporarily blind the user. It is also possible for special applications, like working with strong lasers to protect the eyes of the user to accidental exposure to laser light that could otherwise cause blindness.

Preferably the processing module can adjust the settings of the input module, modify the captured image, and adjust the display engine’s brightness based on the image of the real-world as captured by the input module and optical properties of the adaptable optical filter. This is beneficial so that even when lighting conditions change the user always has the benefit of a overlayed view with clearly visible details. It is an extra benefit when light conditions change for instance when a user goes from bright sunlight outside into a relatively dark room. The input device senses the difference in brightness and the processing module can adjust the overlayed view accordingly, so that for instance in the time a photochromic layer adapts, which can take several minutes, the user still sees an image with clear details. Also, the controllable smart or display filters can be adapted slowly to compensate for the time required by the eye to adapt to different lighting conditions.

Preferably the processing module can construct from the analyzed image, or a sequence of images, a depth map comprising a depth estimation for sectors of the image. Here sectors mean pixels or clusters of pixels. This means the processing unit uses a method, like deep learning or structure from motion. Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. Deep learning is a key technology behind driverless cars, enabling them to recognize a stop sign, or to distinguish a pedestrian from a lamppost. The inventive device uses such methods to get a prediction for the 3D dimension of the real-world. This is especially important when the device is used in accidented terrain, like an urban environment with holes and poles or when driving at night to get a perception of obstructions, like bumps or holes in the road. Preferably the input module also comprises sensors to generate a depth map that covers at least a part of the view of the real-world. These sensors can comprise additional sensors such as a secondary camera, structured light emitter or a LIDAR module. The depth map does not have to cover the complete view of the real-world. For instance, areas in the periphery of the view that are not overlayed with the modified image do not necessarily need to be covered.

Depth estimations from these additional inputs, deep learning or structure from motion techniques can be used to control the distance used for the adaptation of the modified image to the real-world view of the device. For example, adaptation at close range will require a different image transformation as compared to farther away ranges. The depth estimation can be used to automate this process and prevent double vision or distortion.

In a preferred embodiment the depth map can be used to generate a point cloud, planes can be segmented from the point cloud, these planes can be categorized, a ground plane can be identified, and the planes and a deviation from the ground plane can be marked. Such marking can be shown in the modified image. A deviation can also be marked with an acoustic signal in the form of an audio cue or spoken notification, so that the user becomes aware of the danger even when he is not paying attention to the overlayed view. The ground plane is determined by predicting on which plane the user is positioned. This plane will normally be lower than the height of the device. Any deviation from the ground plane can be a potential danger for the user.

Specific color fills or outlines maybe used to selectively highlight such a deviation or specific planes that can be dangerous for a user of the device such as walls, curbs or stairs.

According to the invention additional information related to the image or depth map can be added to the modified image, for instance via the processing module and via the combiner unit. Information such as navigation information or extra information regarding objects, locations, battery life of the device, or device settings and modes, can be added. Navigation information may include written directions, arrows or paths plotted in the environment or a virtual guide that can be followed by the user. Such navigation information includes both high level instructions comparable to instruction provided in a satellite navigation system in a car, as well as low level instructions like short paths around obstacles or road obstructions. Extra information can range from touristic information to warnings about specific traffic situations. Such information can be transferred for instance wireless via a mobile phone.

The invention also relates to a method to enhance visual perception in a see-through device comprising an input module that captures an image of the real-world, a processing module that can analyze and modify the optical properties of the image to create a modified image, and a see-through output module that overlays the view of the real-world with the modified image using a combiner unit and a display engine, characterized in that the processing module analyzes which areas in the image are near or outside the limit of the dynamic range of the human eye and modifies those areas, so that they are redistributed within the dynamic range of the human eye and where the output module selectively overlays the view of the real-world with the areas of the modified image. Preferably the method also provides an enhanced image in a see-through device according to one of the claims.

DESCRIPTION FIGURES

The invention is further explained with the help of the following drawing in which

Figure 1 shows a schematic view of the inventive device,

Figure 2 shows a dynamic range as perceived by the human eye for a low light scene with and without the invented device.

Figure 3 shows an embodiment of the inventive device in the form of smart glasses,

Figure 4 shows a view as obtained with the known device from the prior art,

Figures 5a, b show how the processing module adapts the image from the input module to allign with the view of the real-world, and 5c how a marker can be used to assist with this process.

Figure 6 shows an embodiment of the output module,

Figure 7a, b, c shows a view of the real-world (7a), a modified image from the input module (7b) and the final overlayed image as seen by the user (7c),

Figure 8a, b shows that based on depth information from the input module a ground plane can be identified, and how deviations from this ground plane can be detected,

Figure 9 shows how additional information can be added to the final overlayed image as seen by the user.

Fig. 1 shows a schematic view of a see-through device (1) for enhanced visual perception that comprises an input module (2) that can capture an image of the real- world, a processing module (3) that can analyze and modify the optical properties of the image to create a modified image, and a see-through output module (4) that comprises a combiner unit and a display engine that can overlay the view of the real world with the modified image to create an overlayed view. According to the invention the device (1) is characterized in that the processing module (3) can analyze which areas in the image are near or outside the limit of the dynamic range of the human eye and can modify those areas, so that they are redistributed within the dynamic range of the human eye and where the output module (4) can selectively overlay the view of the real-world with these areas of the modified image.

Preferably the device (1) comprises a control unit (5) that can adjust the properties of modules (2, 3, 4) to accommodate for different lighting conditions in the real world and for different sensitivities of an eye of a user.

Figure 2 illustrates a dynamic range (10) for a low light scene. Here left is low luminance and right is high luminance. The eye without the device (1) has a perceived dynamic range (11). Figure 2 shows that dynamic range (12) beyond the perceivable range (11) in any one view at a given moment in time can be captured by the input module (2), for instance a camera, since the camera can be set to a different sensitivity from the human eye. This dynamic range (12) is than processed by the processing module (3) and redistributed (13) within the perceivable dynamic range (11). The perceivable dynamic range is thus expanded (14), i.e. the eye can now also perceive range (12) through the processing and redistribution (13) into the perceivable range (11). Note that the redistributed range (13) does not necessarily need to be redistributed in the lower section of the perceivable dynamic range (11). Range (13) can also be redistributed in the middle, the right side, or acrass the entire perceivable dynamic range (11), for instance depending on the distribution of luminance levels in range (12). The effect of the redistribution of range (12) into the perceivable dynamic range (11) is seen by the user of the device (1) as a selective illumination of areas in the dynamic range (12).

A similar result can be obtained for dynamic ranges that partly overlap with the perceivable range or outside the perception range for high luminance areas by the human eye, i.e. on the right side of fig. 2. For the latter, the dynamic range outside the perceivable range (11) is dimmed so that it falls again within the range (11).

This device (1) can be configured in embodiments including but not limited to: eye glasses, contact lenses, headsets, helmets, monoculars, binoculars, telescopes, head up displays and windows, for instance of a car.

Fig. 3 shows an example of the electronic see-through device (1) configured as a pair of smart glasses (20) with a monocular camera and display setup. The glasses (20) have an integrated camera (21) as input module (2), an on-board processing module (22) as processing module (3), a display (23) and see-through combiner unit (24).

The control unit (5) can be integrated or separate from the electronic see-through device (1). It may be used to manually operate the camera settings, image processing parameters, brightness of the display and dimming effects, and turn the device on, off or to sleep mode. It may consist of buttons, dials, capacitive or resistive touch elements, microphones, light sensors, eye or hand tracking cameras, or other means by which a user can convey controls. The glasses (20) can be controlled wirelessly or via a capacitive touch strip (25) that functions as input far the build-in control unit (5).

The input module (2) comprises a camera (21) that can capture images in a large variety of light levels (i.e., extreme low light to bright sunlight) and wavelengths such as visible light and infrared, making it usuable as both a day and night vision device.

An infrared illuminator light emitting diode (LED) may be added to the camera to make it useable in low light scenes without any environmental light. The camera settings such as the shutter speed, gain, focus and (infrared) light filter and illuminator may be adjusted dynamically and automatically by the processing module (3) or control module (5) or manually via the control unit (5) to allow for sufficient image quality and performance based on user needs and the real world. In order to facilitate this, images from the camera (21) are processed in the processing module (3,22), which may be integrated or separate from the device, in four sequential stages of which stage two and three can be swapped.

In stage one, entire images, select regions, or pixels are analyzed to measure the luminance levels in the scene and automatically control the settings and characteristics of both the input and output module (2, 4). As the analysis can be conducted on every individual image within a fraction of a second, the device (1) utilizing this methodology can rapidly adapt to changes in luminance levels in the real world.

This analysis may also be used for safety measures. For example, in a low light scene a user may have dilated pupils. When the device (1) is started, the brightness of the display (23) will be set to a low value so as not to risk irritating or damaging the eyes of the user. Brightness may be gradually increased automatically by the processing module or control unit (5) or manually using the control unit (5) to allow the eyes to adapt to scene elements with high luminance levels, such as the headlights from a car.

Due to the pupil size shrinking some scene elements will become too dim to perceive, through the overlayed view the device can bring these elements closer to the luminance level of the headlights, thereby bringing them in the perceivable range.

In stage two image processing filters may be used to adjust or completely filter out values exceeding a threshold or violating a set of requirements. In this step the values of the dynamic range that is easily perceivable by the human eye are filtered. The intensity of such values may be decreased or set to zero (i.e., completely black for digital images). This crucial step distinguishes this methodology from the known device. By filtering out, or lowering the intensity of scene elements that are already perceivable by the eye, the natural vision is not, or to a far lesser extent, compromised by occlusion or double vision in the overlayed view.

Figure 4 shows an example scene to demonstrate the problem of the known device where a misaligned overlay of camera images on top of the real-world view results in double vision.

In stage three the contrast, color and brightness for the overall scene may be adjusted, equalized or redistributed through a series of operations and algorithms.

Figure 5 shows how according to the invention the processing module (3) can adapt the modified image to correct for the difference between the input module's (2) and output module’s (4) specifications and a parallax resulting from the difference between the optical center of the input module (2) and the optical center of the eye of a user based on a controllable distance between the user and the areas observed. Here controllable means that the correction can depend on the distance of an area to the user. The adaptation can include transformations such as: cropping, rotating, scaling as shown in fig 5a, and shifting as shown in fig 5b.

A possible difference in input module (2) specification, i.e. camera lens and image sensor specifications, relative to the output module (4) can for instance depend on a different field of view and aspect ratio. In terms of specifications of the architecture of the system these will be matched as much as possible.

Further, the transformations may be applied to correct for the parallax between the position of the user's eye relative to the position of the camera. Figure 5b shows how the modified image (30) of the camera is shifted along the arrows to match elements (31) of the real world view. Parallax is large for areas close to the user. Areas farther away from the user do not always have to be corrected due to small differences between the modified image (30) and elements in the real world view (31). Figure 5c shows that according to the invention the modified image can contain markers for example dashed edge lines (32, 33) that can be used to facilitate alignment of the modified image (30) with the real-world view (31) as observed through the device, to correct for the difference in component specification and parallax resulting from the difference between the optical center of the input module (2) and the optical center of the eye. These operations are another important distinction from the known device as they further decrease unwanted side effects from misaligned image overlay such as occlusion and double vision.

Also, misalignment may result from a delay between light observed from the real-world view and the light from the same moment in time as captured by the input module (2), processed by the processing module (3) and displayed by the output module (4) which takes more time to reach the eye of a user, Hence this delay should be minimized by optimizing the device so latency is at a minimum and the image capture and display rate is as high is possible. The latency should be no larger than 200ms and the capture and display rate not lower than 20 images per second. Preferably the latency should be no larger than 20ms and image capture and display rate not lower than 60 images per second.

Figure 6 shows how according to the invention the outer surface of the combiner unit (38) of the output module (4) comprises an optical filter (36) that filters light (35) from the real-world view, i.e. the outside world. Preferably the optical filter (38) comprises an adaptable optical filter or a filter that comprises a film or display with semi- transparent areas that can be individually darkened. The optical filter (36) may include a color filter (such as a blue light blocker), (graduated) neutral density filter, a polarization filter, photochromatic material, and coatings such as anti-reflection, anti- fog, UV-blocker, mirror coating or similar. In other embodiments this may also include a smart film or display to allow for electronic control of the light transmission. With these filters (36), light (35) from the real world can be decreased in terms of the luminance to provide light with a lower luminance (37). Next the light (37) enters the combiner element (38). The combiner element (38) provides a see-through view of the real-world from light (37) while at the same time, an overlay (39) with modified images from the display (40). The combiner unit (38) may be based on a free-space combiner, a free-form total internal reflection prism combiner, or a waveguide combiner or any similar optical architecture. The display (40) can consist of three main building blocks: an illumination engine (in case of non-emissive display panels), a display panel or scanner, and optics that can relay the image to the combiner unit (38). The display illumination engine may be based on a laser beam scanner (LBS) display, (active matrix) organic light-emitting diode (AMOLED/OLED) display, liquid crystal display (LCD), liquid crystal on silicon (LCOS) display, micro-light emitting diode (amu-

LED/mu-LED/mu-iLED), digital light processing (DLP) display, or light emitting quantum dots or any similar display paradigm. The brightness of the illumination engine or the emissive display panel can be controlled automatically by the system based on the image analyses conducted in stage one of the processing module (3), or manually by the user via the control module (5) with input (25). Dynamic and rapid control of the brightness of the display is required to permit safe and effective usage in a variety of lighting conditions and increase adaptability to changes in luminance levels of the real world.

Fig. 6 also shows an example of an embodiment of the output module (4) comprising a prescription lens (41). This prescription lens (41) will be placed after the combiner unit (38). The lens (41) may include a single vision lens, bifocal lens, trifocal lens, progressive lens or prism lens.

Finally, the combined light (42) of the real-world view (37) and the overlayed modified image (39) reaches the eye (43).

Since the display illumination engine can only add light to the scene, dark or complete black pixels on the display show up as transparent at the exit pupil of the combiner unit (38). This phenomenon is utilized to selectively overlay the real-world scene (37) with modified images (39). For example, when viewing a real-world scene (37) with both sunlight and shadows in the same space, the device (1) will only overlay the modified images (39) over the shaded areas. As such, the user can more easily see areas in direct sunlight and the shadows at the same time.

Figure 7 shows an example scene to demonstrate how the device (1) according to the invention works. Fig. 7a shows the real-world view as seen by the eye (43) without the device (1). Note that the areas (50, 51) in the shadow appear as black areas, The contrast in these areas (50, 51) is outside the dynamic range of the human eye in any one view at a given moment in time (see also range (12) in fig. 2).

Fig. 7b shows the same view but now seen by the input module (2), i.e. camera (21).

Here, the camera is set so color and contrast differences within the shadows (50, 51) that cannot be seen by the human eye are captured as areas (53, 54). Images from the camera (21) are than processed by the processing module (3) to filter out scene elements of area (55) because contrast and color of these areas (52) in the real-world view are already adequately perceived by the eye. The area (55) is then converted by the processing module (3) to black pixels.

Fig. 7c shows the image that the user of device (1) will see. Since an optical see- through architecture is used, light (39) from display (40) can only be added by the combiner unit (38). Thus, black pixels on the display (40) that do not transmit light,

show up as transparent areas (42) in the overlayed image in fig. 7c. The areas in the shadow (50, 51) are overlayed with modified images (53, 54) modified and adapted from the image of the camera (21). This will result in a selective modification of areas (50,51) with scene elements that are not visible without the device (1), since they are beyond the perceivable dynamic range of the eye. Fig. 7c shows that contrast within areas in direct sunlight (52) and also in the shadows (53, 54) can be perceived at the same time using the device.

Similarly, in a reverse situation with mainly dark areas outside the perceivable dynamic range of the human eye and a few bright spots, the device (1) may overlay the field of view outside the bright spots with a brighter digital modified image. As the human eye adapts to this illumination level the headlights from an oncoming vehicle are well within the contrast range and cause less or even no glare.

The optical filter, smart film or display (36) may be utilized to supplement these effects.

According to the invention based on the image of the real-world as captured by the input module (2) and optical properties of the adaptable filter (36) the processing module (3) can adjust the brightness of the modified image (39). For example, a photochromatic layer (36) which reacts to ultraviolet light from the sun decreases the light (35) from the real-world scene (37). This allows a user to look in areas with direct sunlight more comfortably. Due to the light filtering applied to the real-world view, contrast in areas in the shadows may become not visible to the eye. Using the device with the afore mentioned selective modification method as explained in fig. 7 can increase the perceivable contrast in such regions while leaving other regions unaffected.

Another advantageous example is the following. When a user enters a dark building from a bright outdoor scene a photochromatic material (36) will slowly start transitioning between states, increasing the light transmission rate. This transition process can take several minutes. However, in the inventive device the dynamic operation of the display brightness can be used as compensation during this transition by slowly decreasing the brightness of the display as the photochromatic material becomes more transparent. Hence, the limitations/shortcomings of the photochromatic material (36) can be overcome.

Figure 8 shows that the processing module (3) can construct from the analyzed image of the input module (2) or a sequence of images, a depth map comprising a depth estimation for sectors of the image. Such sectors can comprise one pixel or a group of pixels. This can be used for creating a point cloud and marking depth-based obstacles, i.e. obstacles with a certain height. The scene of fig. 8 is viewed by the eye (43) and the camera (21) simultaneously. Analyzed images from the camera (21) are than processed to obtain a depth map of the scene of fig. 8. The depth map may also be generated through movement of a structure in successive images, structured light (e.g, an infrared dot pattern projected in the environment), a stereo camera or LiDAR input.

Fig. 8a, b show how according to the invention planes can be segmented from a point cloud generated from the depth map, a ground plane (61) can be identified and a deviation (62, 63) from this plane (61) can be marked.

Distances beyond a particular threshold are not segmented or identified, i.e. area (60) in fig. 8a. Such far away deviations/obstacles pose no direct threat for the user. Using a plane segmentation algorithm, a ground plane (61) for distances below the threshold is identified. This plane (61) is the surface, the user is presumed to be positioned on.

Subsequently all deviations (62, 83) from this ground plane (61) are identified and marked, for instance by color coding with a fill or outline {see fig. 8b). All scene elements except the marked deviations (62, 63) are converted to black pixels (64) that will not be shown in the overlay (42). Since an optical see-through architecture is used this will result in a selective showing of depth-based markers, i.e. color coded or outlined scene elements (62, 63) that can help the user identify obstacles. The device may also mark all identified surfaces with a set of color fills or outlines to promote spatial awareness for the user of the device. Preferably the deviations (62, 63) are also marked with an acoustic signal in case the user does not pay attention to the overlayed image (42).

Figure 9 shows how additional information related to the device (1), image (42) or depth map can be added through the processing module (3) or the combiner unit (38).

Such information can contain details such as device settings, modes, battery status, information about objects or scenes. Figure 9 shows an example of a navigation marker (65) plotted in the real-world through the overlayed image of the device.

Claims

CONCLUSIONS

1. A see-through apparatus for enhanced visual perception comprising an input module capable of capturing an image of reality, a processing module capable of analyzing and modifying optical properties of the image to create a modified image, and a see-through output module comprising a combining unit and a display engine capable of overlapping a view of reality with the modified image to create an overlapping view, characterized in that the processing module can analyze which areas in the image are close to or outside the limit of the dynamic range of the human eye and modify these areas so that they are redistributed within the dynamic range of the human eye, and wherein the output module can selectively overlap the view of reality with these areas of the modified image.

2. Apparatus according to claim 1, characterised in that the apparatus comprises a control unit which can adapt properties of modules to different lighting conditions in reality and to different sensitivities of a user's eye.

3. Apparatus according to claim 2, characterised in that the apparatus comprises a camera directed at the user's eye to predict a light adaptation state of the eye by adjusting the size of the pupil of the eye to preset user-specific values.

4. Apparatus according to one or more of the preceding claims, characterized in that the processing module can adjust the modified image to correct for the difference between specifications of the input and output modules and a parallax due to a difference between the optical center of the input module and the optical center of the user's eye.

5. Apparatus according to claim 4, characterised in that the modified image comprises a marking which can be used to facilitate the modification of the modified image.

6. Apparatus according to any one of the preceding claims, characterized in that an outer surface of the combining unit of the output module comprises an optical filter capable of filtering light from the reality.

7. Apparatus according to claim 6, characterised in that the optical filter comprises an adjustable optical filter.

8. Apparatus according to claim 7, characterised in that the optical filter comprises a film or screen with semi-transparent areas which can be individually darkened.

9. Apparatus according to claim 7 or 8, characterized in that the processing module can adjust settings of the input module, change the captured image and adjust the brightness of the display machine based on the image of reality as captured by the input module and optical properties of the adjustable optical filter.

10. Apparatus according to any one of the preceding claims, characterised in that the processing module can compile from the analysed image or a series of images a depth map with a depth estimate for sectors of the image.

11. Apparatus according to claim 10, characterized in that the input module comprises sensors to generate a depth map covering at least a part of the view of reality.

12. Apparatus according to claim 10 or 11, characterised in that the depth map can be used to generate a point cloud, that surfaces from the point cloud can be segmented, that these surfaces can be categorised, that a ground plane can be identified, and the surfaces and a deviation from the ground plane can be marked.

13. Apparatus according to any one of the preceding claims, characterised in that additional information relating to the image or the depth map can be added to the modified image.

14. A method for enhanced visual perception in a see-through apparatus, comprising an input module that captures an image of reality, a processing module that can analyze and modify optical properties of the image to create a modified image, and a see-through output module that overlays the view of reality with the modified image using a combining unit and a display engine, characterized in that the processing module analyzes which areas in the image are close to or outside the limit of the dynamic range of the human eye and modifies those areas so that they are redistributed within the dynamic range of the human eye and wherein the output module selectively overlays the areas of the modified image onto the view of reality.

15. Method for enhanced visual perception using an apparatus according to any one of claims 1 to 13.