WO2024216573A1

WO2024216573A1 - Composite image generation using motion information

Info

Publication number: WO2024216573A1
Application number: PCT/CN2023/089434
Authority: WO
Inventors: Boen SONG; Ke HE; Fan XIAO
Original assignee: Qualcomm Inc
Current assignee: Qualcomm Inc
Priority date: 2023-04-20
Filing date: 2023-04-20
Publication date: 2024-10-24
Anticipated expiration: 2025-10-20

Abstract

Systems and techniques are described herein for generating a composite image. For instance, a method for generating a composite image is provided. The method may include obtaining a first short-exposure image associated with a first exposure duration; obtaining a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; obtaining a second short-exposure image associated with the first exposure duration; obtaining a second long-exposure image associated with the second exposure duration; comparing the first short-exposure image with the second short-exposure image; comparing the first long-exposure image with the second long-exposure image; determining a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and generating a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

Description

COMPOSITE IMAGE GENERATION USING MOTION INFORMATION

TECHNICAL FIELD

The present disclosure generally relates to generating a composite image. For example, aspects of the present disclosure include systems and techniques for obtaining multiple images of a scene and generating a composite image including respective pixels from each of two or more of the multiple images. Some aspects relate to determining a motion map indicative of motion within the scene and determining which pixels to select from each of the multiple images based on the motion map.

BACKGROUND

A camera is a device that receives light and captures image frames, such as still images or video frames, using an image sensor. Cameras can be configured with a variety of image-capture and/or image-processing settings to alter the appearance of images captured thereby. Some camera settings are determined and applied before and/or while an image is captured, such as ISO, exposure time (also referred to as exposure duration or shutter speed) , aperture size, (also referred to as f/stop) , focus, and gain (including analog and/or digital gain) , among others. Moreover, some camera settings can be configured for post-processing of an image, such as alterations to contrast, brightness, saturation, sharpness, levels, curves, and colors, among others. In some cases, a camera can capture multiple images of a scene using different image-capture parameters and can combine the captured images into a single image frame.

SUMMARY

The following presents a simplified summary relating to one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the following summary be considered to identify key or critical elements relating to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts relating to one or more aspects relating to the mechanisms disclosed herein in a simplified form to precede the detailed description presented below.

Systems and techniques are described for generating a composite image. According to at least one example, a method is provided for generating a composite image. The method includes: obtaining a first short-exposure image associated with a first exposure duration; obtaining a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; obtaining a second short-exposure image associated with the first exposure duration; obtaining a second long-exposure image associated with the second exposure duration; comparing the first short-exposure image with the second short-exposure image; comparing the first long-exposure image with the second long-exposure image; determining a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and generating a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

In another example, an apparatus for generating a composite image is provided that includes at least one memory and at least one processor (e.g., configured in circuitry) coupled to the at least one memory. The at least one processor configured to: obtain a first short-exposure image associated with a first exposure duration; obtain a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; obtain a second short-exposure image associated with the first exposure duration; obtain a second long-exposure image associated with the second exposure duration; compare the first short-exposure image with the second short-exposure image; compare the first long-exposure image with the second long-exposure image; determine a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and generate a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: obtain a first short-exposure image associated with a first exposure duration; obtain a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; obtain a second short-exposure image associated with the first exposure duration; obtain a second long-exposure image associated with the second exposure duration; compare the first short-exposure image with the second short- exposure image; compare the first long-exposure image with the second long-exposure image; determine a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and generate a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

In another example, an apparatus for generating a composite image is provided. The apparatus includes: one or more means for obtaining a first short-exposure image associated with a first exposure duration; means for obtaining a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; means for obtaining a second short-exposure image associated with the first exposure duration; means for obtaining a second long-exposure image associated with the second exposure duration; means for comparing the first short-exposure image with the second short-exposure image; means for comparing the first long-exposure image with the second long-exposure image; means for determining a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and means for generating a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

In some aspects, one or more of the apparatuses described herein is, can be part of, or can include a mobile device (e.g., a mobile telephone or so-called “smart phone” , a tablet computer, or other type of mobile device) , an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device) , a vehicle (or a computing device or system of a vehicle) , a smart or connected device (e.g., an Internet-of-Things (IoT) device) , a wearable device, a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television) , a robotics device or system, or other device. In some aspects, each apparatus can include an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each apparatus can include one or more displays for displaying one or more images, notifications, and/or other displayable data. In some aspects, each apparatus can include one or more speakers, one or more light-emitting devices, and/or one or more microphones. In some aspects, each apparatus can include one or more sensors. In some cases, the one or more sensors can be used for determining a location of the apparatuses, a state of the apparatuses (e.g., a tracking state, an operating state, a temperature, a humidity level, and/or other state) , and/or for other purposes.

This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative examples of the present application are described in detail below with reference to the following figures:

FIG. 1 is a block diagram illustrating an example architecture of an image-processing system according to various aspects of the present disclosure;

FIG. 2 is a diagram illustrating multiple images with different exposures used to create a composite image according to various aspects of the present disclosure;

FIG. 3 is a block diagram illustrating a system for generating a composite image, according to various aspects of the present disclosure;

FIG. 4 is an example visual representation of a short-and-long-exposure motion map, according to various aspects of the present disclosure;

FIG. 5 is an example of visual representation of a highlight map, according to various aspects of the present disclosure;

FIG. 6 is an example visual representation of a short-exposure motion map, according to various aspects of the present disclosure;

FIG. 7 is an example visual representation of a long-exposure motion map, according to various aspects of the present disclosure;

FIG. 8 is an example visual representation of a reference motion map, according to various aspects of the present disclosure;

FIG. 9 is an example visual representation of a final motion map, according to various aspects of the present disclosure;

FIG. 10 is an example visual representation of a fusion map, according to various aspects of the present disclosure;

FIG. 11 is a flow diagram illustrating example process for generating a composite image, in accordance with aspects of the present disclosure;

FIG. 12 illustrates an example computing-device architecture of an example computing device which can implement the various techniques described herein.

DETAILED DESCRIPTION

Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

The ensuing description provides example aspects only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the exemplary aspects will provide those skilled in the art with an enabling description for implementing an exemplary aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the application as set forth in the appended claims.

The terms “exemplary” and/or “example” are used herein to mean “serving as an example, instance, or illustration. ” Any aspect described herein as “exemplary” and/or “example” is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

In photography and videography, a technique referred to as high dynamic range (HDR) allows the dynamic range of image frames captured by a camera to be increased beyond the native capability of the camera. In this context, a dynamic range refers to the range of luminosity between the brightest area and the darkest area of the scene or image frame. For example, a high dynamic range means there is large variation in light levels within a scene or an image frame.

HDR can involve capturing multiple images of a scene using different image-capture parameters (e.g., a short-exposure image and a long-exposure image) and combining the multiple images to generate a composite image. In the present disclosure, the term “exposure, ” relating to an image captured by a camera, refers to the amount of light per unit area that reaches a photographic film, or in modern cameras, an electronic image sensor (e.g., including an array of photodiodes) . The exposure is based on certain camera settings such as, for example, shutter speed and/or lens aperture, as well as the luminance of the scene being photographed. Many cameras are equipped with an automatic exposure or “auto exposure” mode, where the exposure settings (e.g., shutter speed, lens aperture, etc. ) of the camera may be automatically adjusted based on the luminance of a scene or subject being photographed. In some cases, an automatic exposure control (AEC) engine can perform AEC to determine exposure settings for an image sensor. In the present disclosure, the terms “exposure duration” and “exposure time” may refer to a duration of time during which the electronic image sensor is exposed to light (e.g., while the electronic image sensor is capturing an image) .

In one example, an imaging device can generate a composite image by combining a short-exposure image captured with a short exposure time and a long-exposure image captured with a long exposure time (that is longer than the short exposure time) . For instance, because short-exposure images are generally dark, they generally preserve the most detail in the bright areas (e.g., highlights) of a photographed scene. Long-exposure images are generally brighter than short-exposure images, and may be overexposed (e.g., too bright to make out details) in the highlight portions (bright areas) of the scene. Because long-exposure images generally include bright portions, they may preserve detail in the dark areas (e.g., shadows) of a photographed scene. To generate a composite image, an imaging device may, for example, use portions of the short-exposure image to depict bright areas (e.g., highlights) of the photographed scene and use portions of the long-exposure image depicting dark areas (e.g., shadows) of the scene. The combination of image frames with different exposure times can result in an image with a dynamic range higher than that of each individual image frame captured and combined to form the composite image. HDR is a feature often used by electronic devices, such as cameras, smartphones, and mobile devices, for various purposes. For example, in some cases, a smartphone can use HDR to achieve a better image quality or an image quality similar to the image quality achieved by a digital single-lens reflex (DSLR) camera.

However, if there is motion in the scene, pixels of the respective images may be different between the multiple images. For example, while a long-exposure image is being captured, an object in the scene may move. The object may be blurred in the long-exposure image because the object may move while the long-exposure image is being captured. In contrast, the same object may move less during the capturing of the short-exposure image because the short-exposure image took less time to capture than the long-exposure image. Thus, the blurred pixels of the long-exposure image may not be the same as the corresponding pixels of the short-exposure image. Further, one or the other of the short-exposure image and the long-exposure image may be captured first (e.g., according to a staggered HDR (SHDR) technique) and the object may be at a different location within the scene by the time the second image is captured.

Systems, apparatuses, methods (also referred to as processes) , and computer-readable media (collectively referred to herein as “systems and techniques” ) are described herein for generating a composite image. The systems and techniques described herein may determine a motion map based on a comparison between two short-exposure images and/or two long-exposure images and generate the composite image based on the motion map.

For instance, to address motion in the context of HDR imaging, images may be combined not only based on brightness, but based on motion within the scene. For example, a motion map that indicates motion in the scene may be determined based on pixels that are different between the images. When combining images, the systems and techniques may use the motion map to determine which pixels to take from which of the images. For example, the systems and techniques can select pixels in which motion between exposures has been detected (e.g., as indicated by a motion map) from a short-exposure image for inclusion in a composite image. The pixels where motion has been detected may be clearer in the short-exposure image than in the long-exposure image because the moving object may have moved less during the capture of the short-exposure image than during the capture of the long-exposure image. Thus, for the pixels where motion has been detected, the systems and techniques may use the pixels from the short-exposure image when generating the composite image.

In some cases, the motion map may be determined based on a comparison between the short-exposure image and the long-exposure image that are combined into the composite image. However, determining a motion map by comparing a short-exposure image and a long-exposure image may be complicated by the fact that short-exposure images may exhibit more noise than long-exposure image. In the present disclosure, the term “noise, ” when referring to images, may refer to random variation in brightness and/or color information due to electronic noise and/or inconsistencies in the image-capture device. Noise in the short-exposure image may generate false positives when comparing a short-exposure image to a long-exposure image to determine a motion map. Further, in some situations, brightness compensation and/or digital gain can exacerbate the issue by increasing noise in the short-exposure image. A motion map with false positives can result in poor composite-image quality (e.g., composite images exhibiting noise in static areas of the composite image) . Motion may be detected by selecting positions in which a pixel difference between a short-exposure image and a long-exposure image is higher than a threshold. If a pixel difference that was caused by motion is not detected because the pixel difference is below the threshold, the result is a false negative. A motion map with false negatives can result in poor composite-image quality (e.g., composite images exhibiting ghosting in moving areas of the composite image) .

The systems and techniques may thus generate a short-exposure motion map based on a comparison between two short-exposure images. Because the two short-exposure images may have similar gains (e.g., analog and/or digital gain) and similar exposure times, the two short-exposure images may exhibit similar noise. Because the two short-exposure images may exhibit similar noise, when the two short-exposure images are compared to generate the short-exposure motion map, the similar noise may be cancelled out. Additionally, or alternatively, the similar noise may be removed from the comparison based on thresholds for the comparison, for example, the thresholds for identifying motion may be set tighter (e.g., by decreasing the size of the threshold) to exclude the noise. With the similar noise cancelled out, or removed through thresholds, the short-exposure motion map generated based on the two short-exposure images may include fewer false positives and/or false negatives than a motion map based on a comparison between a short-exposure image and a long-exposure image.

Similarly, the systems and techniques may generate a long-exposure motion map based on a comparison between two long-exposure images. The two long-exposure images may have similar gains (e.g., analog and/or digital gain) and similar exposure times. Thus, the two long-exposure images may exhibit similar noise. Similar to the comparison of the two short-exposure images, in comparing the two long-exposure images to generate the long-exposure motion map, the similar noise may be cancelled out. Additionally, or alternatively, the similar noise may be removed from the comparison based on thresholds for the comparison. With the similar noise cancelled out, or removed through thresholds, the long-exposure motion map generated based on the two long-exposure images may include fewer false positives and/or false negatives than a motion map based on a comparison between a short-exposure image and a long-exposure image.

In some cases, the systems and techniques may generate a reference motion map based on the short-exposure motion map and the long-exposure motion map. Further, in some cases, the systems and techniques may generate a short-and-long-exposure motion map by comparing a short-exposure image and a long-exposure image. The systems and techniques may generate a fusion map based on the reference motion map and the short-and-long-exposure motion map. The systems and techniques may further base the fusion map on a highlight map indicative of bright and/or dark areas in the images. The systems and techniques may generate a composite image based on the fusion map.

In any case, by generating a short-exposure motion map and/or a long-exposure motion map, the systems and techniques may use a motion map that has fewer false positives and/or false negatives than a motion map based on a comparison between a short-exposure image and a long-exposure image. The systems and techniques may use the short-exposure motion map and/or the long-exposure motion map to generate composite images that are superior to other composite images generated by other HDR techniques. For example, the composite images generated by the systems and techniques may exhibit less noise in static areas of the composite images and/or less ghosting in moving areas of the composite images than the other composite images of the other HDR techniques.

Various aspects of the application will be described with respect to the figures below.

FIG. 1 is a block diagram illustrating an example architecture of an image-processing system 100 according to various aspects of the present disclosure. The image-processing system 100 includes various components that are used to capture and process images, such as an image of a scene 106. The image-processing system 100 can capture image frames (e.g., still images or video frames) . In some cases, the lens 108 and image sensor 118 (which may include an analog-to-digital converter (ADC) ) can be associated with an optical axis. In one illustrative example, the photosensitive area of the image sensor 118 (e.g., the photodiodes) and the lens 108 can both be centered on the optical axis.

In some examples, the lens 108 of the image-processing system 100 faces a scene 106 and receives light from the scene 106. The lens 108 bends incoming light from the scene toward the image sensor 118. The light received by the lens 108 then passes through an aperture of the image-processing system 100. In some cases, the aperture (e.g., the aperture size) is controlled by one or more control mechanisms 110. In other cases, the aperture can have a fixed size.

The one or more control mechanisms 110 can control exposure, focus, and/or zoom based on information from the image sensor 118 and/or information from the image processor 124. In some cases, the one or more control mechanisms 110 can include multiple mechanisms and components. For example, the control mechanisms 110 can include one or more exposure-control mechanisms 112, one or more focus-control mechanisms 114, and/or one or more zoom-control mechanisms 116. The one or more control mechanisms 110 may also include additional control mechanisms besides those illustrated in FIG. 1. For example, in some cases, the one or more control mechanisms 110 can include control mechanisms for controlling analog gain, flash, HDR, depth of field, and/or other image capture properties.

The focus-control mechanism 114 of the control mechanisms 110 can obtain a focus setting. In some examples, focus-control mechanism 114 stores the focus setting in a memory register. Based on the focus setting, the focus-control mechanism 114 can adjust the position of the lens 108 relative to the position of the image sensor 118. For example, based on the focus setting, the focus-control mechanism 114 can move the lens 108 closer to the image sensor 118 or farther from the image sensor 118 by actuating a motor or servo (or other lens mechanism) , thereby adjusting the focus. In some cases, additional lenses may be included in the image-processing system 100. For example, the image-processing system 100 can include one or more microlenses over each photodiode of the image sensor 118. The microlenses can each bend the light received from the lens 108 toward the corresponding photodiode before the light reaches the photodiode.

In some examples, the focus setting may be determined via contrast detection autofocus (CDAF) , phase detection autofocus (PDAF) , hybrid autofocus (HAF) , or some combination thereof. The focus setting may be determined using the control mechanism 110, the image sensor 118, and/or the image processor 124. The focus setting may be referred to as an image capture setting and/or an image processing setting. In some cases, the lens 108 can be fixed relative to the image sensor and the focus-control mechanism 114.

The exposure-control mechanism 112 of the control mechanisms 110 can obtain an exposure setting. In some cases, the exposure-control mechanism 112 stores the exposure setting in a memory register. Based on the exposure setting, the exposure-control mechanism 112 can control a size of the aperture (e.g., aperture size or f/stop) , a duration of time for which the aperture is open (e.g., exposure time or shutter speed) , a duration of time for which the sensor collects light (e.g., exposure time or electronic shutter speed) , a sensitivity of the image sensor 118 (e.g., ISO speed or film speed) , analog gain applied by the image sensor 118, or any combination thereof. The exposure setting may be referred to as an image capture setting and/or an image processing setting.

The zoom-control mechanism 116 of the control mechanisms 110 can obtain a zoom setting. In some examples, the zoom-control mechanism 116 stores the zoom setting in a memory register. Based on the zoom setting, the zoom-control mechanism 116 can control a focal length of an assembly of lens elements (lens assembly) that includes the lens 108 and one or more additional lenses. For example, the zoom-control mechanism 116 can control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanism) to move one or more of the lenses relative to one another. The zoom setting may be referred to as an image capture setting and/or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly may include a focusing lens (which can be lens 108 in some cases) that receives the light from the scene 106 first, with the light then passing through an afocal zoom system between the focusing lens (e.g., lens 108) and the image sensor 118 before the light reaches the image sensor 118. The afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses of equal or similar focal length (e.g., within a threshold difference of one another) with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom-control mechanism 116 moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses. In some cases, zoom-control mechanism 116 can control the zoom by capturing an image from an image sensor of a plurality of image sensors (e.g., including image sensor 118) with a zoom corresponding to the zoom setting. For example, the image-processing system 100 can include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, the zoom-control mechanism 116 can capture images from a corresponding sensor.

The image sensor 118 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures an amount of light that eventually corresponds to a particular pixel in the image produced by the image sensor 118. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes can be covered in color filters, and may thus measure light matching the color of the filter covering the photodiode. Various color filter arrays can be used such as, for example and without limitation, a Bayer color filter array, a quad color filter array (QCFA) , and/or any other color filter array.

In some cases, the image sensor 118 may alternately or additionally include opaque and/or reflective masks that block light from reaching certain photodiodes, or portions of certain photodiodes, at certain times and/or from certain angles. In some cases, opaque and/or reflective masks may be used for phase detection autofocus (PDAF) . In some cases, the opaque and/or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., an IR cut filter, a UV cut filter, a band-pass filter, low-pass filter, high-pass filter, or the like) . The image sensor 118 may also include an analog gain amplifier to amplify the analog signals output by the photodiodes and/or an analog to digital converter (ADC) to convert the analog signals output of the photodiodes (and/or amplified by the analog gain amplifier) into digital signals. In some cases, certain components or functions discussed with respect to one or more of the control mechanisms 110 may be included instead or additionally in the image sensor 118. The image sensor 118 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixel sensor (APS) , a complimentary metal-oxide semiconductor (CMOS) , an N-type metal-oxide semiconductor (NMOS) , a hybrid CCD/CMOS sensor (e.g., sCMOS) , or some other combination thereof.

The image processor 124 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 128) , one or more host processors (including host processor 126) , and/or one or more of any other type of processor discussed with respect to the computing-device architecture 1200 of FIG. 12. The host processor 126 can be a digital signal processor (DSP) and/or other type of processor. In some implementations, the image processor 124 is a single integrated circuit or chip (e.g., referred to as a system-on-chip or SoC) that includes the host processor 126 and the ISP 128. In some cases, the chip can also include one or more input/output ports (e.g., input/output (I/O) ports 130) , central processing units (CPUs) , graphics processing units (GPUs) , broadband modems (e.g., 3G, 4G or LTE, 5G, etc. ) , memory, connectivity components (e.g., BluetoothTM, Global Positioning System (GPS) , etc. ) , any combination thereof, and/or other components. The I/O ports 130 can include any suitable input/output ports or interface according to one or more protocol or specification, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General-Purpose Input/Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-performance Bus (AHB) bus, any combination thereof, and/or other input/output port. In one illustrative example, the host processor 126 can communicate with the image sensor 118 using an I2C port, and the ISP 128 can communicate with the image sensor 118 using an MIPI port.

The image processor 124 may perform a number of tasks, such as de-mosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC) , CDAF, PDAF, automatic white balance, combining of image frames to form a composite image, image recognition, object recognition, feature recognition, receipt of inputs, managing outputs, managing memory, or some combination thereof. The image processor 124 may store image frames and/or processed images in random-access memory (RAM) 120, read-only memory (ROM) 122, a cache, a memory unit, another storage device, or some combination thereof.

Various input/output (I/O) devices 132 may be connected to the image processor 124. The I/O devices 132 can include a display screen, a keyboard, a keypad, a touchscreen, a trackpad, a touch-sensitive surface, a printer, any other output devices, any other input devices, or any combination thereof. In some cases, a caption may be input into the image-processing device 104 through a physical keyboard or keypad of the I/O devices 132, or through a virtual keyboard or keypad of a touchscreen of the I/O devices 132. The I/O devices 132 may include one or more ports, jacks, or other connectors that enable a wired connection between the image-processing system 100 and one or more peripheral devices, over which the image-processing system 100 may receive data from the one or more peripheral device and/or transmit data to the one or more peripheral devices. The I/O devices 132 may include one or more wireless transceivers that enable a wireless connection between the image-processing system 100 and one or more peripheral devices, over which the image-processing system 100 may receive data from the one or more peripheral device and/or transmit data to the one or more peripheral devices. The peripheral devices may include any of the previously-discussed types of the I/O devices 132 and may themselves be considered I/O devices 132 once they are coupled to the ports, jacks, wireless transceivers, or other wired and/or wireless connectors.

As shown in FIG. 1, a vertical dashed line divides the image-processing system 100 of FIG. 1 into two portions that represent the image-capture device 102 and the image-processing device 104, respectively. The image-capture device 102 includes the lens 108, control mechanisms 110, and the image sensor 118. The image-processing device 104 includes the image processor 124 (including the ISP 128 and the host processor 126) , the RAM 120, the ROM 122, and the I/O device 132. In some cases, certain components illustrated in the image-capture device 102, such as the ISP 128 and/or the host processor 126, may be included in the image-capture device 102. In some examples, the image-processing system 100 can include one or more wireless transceivers for wireless communications, such as cellular network communications, 802.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof.

The image-processing system 100 can be part of, or implemented by, a single computing device or multiple computing devices. In some examples, the image-processing system 100 can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc. ) , a telephone system (e.g., a smartphone, a cellular telephone, a conferencing system, etc. ) , a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a game console, an XR device (e.g., an HMD, smart glasses, etc. ) , an IoT (Internet-of-Things) device, a smart wearable device, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device (s) .

The image-capture device 102 and the image-processing device 104 can be part of the same electronic device or different electronic devices. In some implementations, the image-capture device 102 and the image-processing device 104 can be different devices. For instance, the image- capture device 102 can include a camera device and the image-processing device 104 can include a computing device, such as a mobile device, a desktop computer, a smartphone, a smart television, a game console, or other computing device.

While the image-processing system 100 is shown to include certain components, one of ordinary skill will appreciate that the image-processing system 100 can include more components than those shown in FIG. 1. The components of the image-processing system 100 can include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image-processing system 100 can include and/or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and/or other suitable electronic circuits) , and/or can include and/or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. The software and/or firmware can include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of the electronic device implementing the image-processing system 100.

In some examples, the computing-device architecture 1200 shown in FIG. 12 and further described below can include the image-processing system 100, the image-capture device 102, the image-processing device 104, or a combination thereof.

In some examples, the image-processing system 100 can create a composite image using multiple images captured according to different image-capture parameters. For example, the image-processing system 100 can create a composite image using a short-exposure image and a long-exposure image. In some cases, image-capture device 102 can capture the short-exposure image and the long-exposure image write the different images from one or more camera frontend engines to a memory device, such as a RAM 120 or any other memory device. Image processor 124 can then retrieve the images and combine them into a single composite image.

In the present disclosure, the term “combine, ” and like terms, with reference to images or image data, may refer to any suitable techniques for using information (e.g., pixels) from two or more images to generate an image. For example, pixels from a first image and pixels from a second image may be combined to generate a composite image. In such cases some of the pixels of the composite image may be from the first image and others of the pixels of the composite image may be from the second image. In some cases, some of the pixels from the first image and the second image may be merged, fused, or blended. For example, color and/or intensity values for pixels of the composite image may be based on respective pixels from both the first image and the second image. For instance, a given pixel of the composite image may be based on an average, or a weighted average, between a corresponding pixel of the first image and a corresponding pixel of the second image (e.g., the corresponding pixels of the first image and the second image may be blended) . As one example, a central region of a first image may be included in a composite image. Further, an outer region of a second of a second image may be included in the composite image. Pixels surrounding the central region in the composite image may be based on weighted averages between corresponding pixels of the first image and corresponding pixels of the second image. In other words, pixels of the first image surrounding the central region may be merged, fused, or blended with pixels of the second image inside the outer region.

FIG. 2 is a diagram illustrating multiple images with different exposures used to create a composite image 210 according to various aspects of the present disclosure. In particular, FIG. 2 illustrates a short-exposure image 202, a long-exposure image 206, and a composite image 210 generated by combining together the short-exposure image 202 and the long-exposure image 206. The short-exposure image 202 includes under-exposed pixels 204, and the long-exposure image 206 includes over-exposed pixels 208. Composite image 210 includes pixels from short-exposure image 202 and pixels from long-exposure image 206. In particular, composite image 210 includes pixels from short-exposure image 202 in place of over-exposed pixels 208 and pixels from long-exposure image 206 in place of pixels from under-exposed pixels 204.

FIG. 3 is a block diagram illustrating a system 300 for generating a composite image 352, according to various aspects of the present disclosure. System 300 may be implemented in an image-processing device (e.g., image-processing device 104 of FIG. 1, for example, by an image processor 124, host processor 126, and/or ISP 128 of image-processing device 104) .

System 300 may receive long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308. In the present disclosure, the term “image” may be used to refer to an image at multiple different stages between when light is captured by an image sensor and when data representative of the captured light is stored, displayed, and/or transmitted. For example, the term “image” may refer to data generated as light is captured by an image sensor, data representative of the light as transmitted between the image sensor and an image-processing device, and/or data representative of the light as stored, presented, and/or transmitted after processing by the image-processing device. Long-exposure image 302 short-exposure image 304, short-exposure image 306, and long-exposure image 308 may be data representative of light captured by an image sensor (e.g., as received by an image-processing device) .

Long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308 may all represent the same scene. Long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308 may be captured within a short time (e.g., less than 10 seconds or less than 1 second) of one another. In some cases, long-exposure image 302 and short-exposure image 304 may be captured prior to the capturing of short-exposure image 306 and long-exposure image 308. For example, long-exposure image 302 and short-exposure image 304 may be captured as preview images before a user instructs an image-capture device to capture short-exposure image 306 and long-exposure image 308. As another example, a user may press a shutter button, instructing an image-capture device to capture an image and all of long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308, may be captured in quick succession (e.g., within one second) . As another example, long-exposure image 302 and short-exposure image 304 may be captured first in a series of images of a video and short-exposure image 306 and long-exposure image 308 may be captured second in the series of images.

In some cases, long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308 may be spatially correlated (e.g., pixels of long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308 may be aligned based on a movement of an image-capture device between capturing long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308) . For example, long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308 may be aligned according to a global-motion detection technique.

Short-exposure image 304 and short-exposure image 306 may be captured over a shorter exposure time than the exposure time during which long-exposure image 302 and long-exposure image 308 are captured. Short-exposure image 304 and short-exposure image 306 may be captured using similar image-capture parameters (e.g., similar exposure times, lens apertures, gains (including digital and analog gain) , etc. ) . Also, long-exposure image 302 and long-exposure image 308 may be captured using similar image-capture parameters.

Pre-processor 310 may process long-exposure image 302 resulting in processed long-exposure image 318, pre-processor 312 may process short-exposure image 304 resulting in processed short-exposure image 320, pre-processor 314 may process short-exposure image 306 resulting in processed short-exposure image 322, and pre-processor 316 may process long-exposure image 308 resulting in processed long-exposure image 324. Each of pre-processor 310, pre-processor 312, pre-processor 314, and pre-processor 316 may perform black-level deduction on long-exposure image 302, short-exposure image 304, short-exposure image 306, and long-exposure image 308 respectively. For example, black-level deduction may include compensating for the impact of the sensitization component generating dark current based on temperature and/or optical component defects. Additionally, or alternatively, black-level deduction cause values of the images to be in the linear domain. Black-level deduction may be performed by, for example, measuring intensity values in all black environment (e.g., in a calibration phase) and deducting the measured intensities from measured intensity values for each sensor to obtain a linear raw image. Pre-processor 312 and pre-processor 314 may perform exposure-ratio compensation on short-exposure image 304 and short-exposure image 306 respectively. For example, exposure-ration compensation may include scaling the brightness of pixels in short exposure image 304 and short-exposure image 306 based on the brightness of pixels in long-exposure image 302 and long-exposure images 308 respectively. For example, exposure-ration compensation may include aligning the brightness of pixels in short exposure image 304 and short-exposure image 306 with the brightness of pixels in long-exposure image 302 and long-exposure images 308 respectively. Additionally, or alternatively, pre-processor 312 and pre-processor 314 may perform low-pass filtering on under-exposed pixels 204 and short-exposure image 306 respectively. Low pass filtering short-exposure image 304 and short-exposure image 306 may remove some of the noise of short-exposure image 304 and short-exposure image 306.

Motion detector and highlight detector 330 may receive processed short-exposure image 322 from pre-processor 314 and processed long-exposure image 324 from pre-processor 316. Motion detector and highlight detector 330 may compare processed short-exposure image 322 with processed long-exposure image 324 to generate short-and-long-exposure motion map 336. Short-and-long-exposure motion map 336 may be based on differences between processed short-exposure image 322 and processed long-exposure image 324.

Short-and-long-exposure motion map 336 may include a number of indications of differences between pixels of processed short-exposure image 322 and processed long-exposure image 324. For example, short-and-long-exposure motion map 336 may include a difference value for each pixel in processed short-exposure image 322, each difference value indicative of a degree of difference between a respective pixel of processed short-exposure image 322 and a corresponding pixel of processed long-exposure image 324. Each of the difference values may correspond to two pixels (e.g., a pixel of processed short-exposure image 322 and a corresponding pixel of processed long-exposure image 324) . As such, each of the difference values of short-and-long-exposure motion map 336 may be referred to as a pixel. For descriptive purpose, short-and-long-exposure motion map 336 may be visualized (e.g., with black and dark gray representing low degrees of difference and white and light gray representing high degrees of difference) .

FIG. 4 is an example visual representation of a short-and-long-exposure motion map 400, according to various aspects of the present disclosure. In short-and-long-exposure motion map 400, and in other motion maps included herein, dark pixels represent a relatively low degree of difference between images and light pixels represent a relatively high degree of difference between images. Short-and-long-exposure motion map 400 may be an example of short-and-long-exposure motion map 336 of FIG. 3.

Notably, in short-and-long-exposure motion map 400, there are many false positives. For example, the buildings in the background of short-and-long-exposure motion map 400 are not moving. But, based on noise in the short-exposure image used to generate short-and-long-exposure motion map 400, there are differences between the short-exposure image and the long-exposure image. The differences are interpreted in short-and-long-exposure motion map 400 as motion.

Returning to FIG. 3, additionally, motion detector and highlight detector 330 may generate a highlight map 338. Highlight map 338 may be based on luminance in regions of processed short-exposure image 322 and/or processed long-exposure image 324. Highlight map 338 may be useful to determine which pixels of composite image 352 to select from processed short-exposure image 322 and which to select from processed long-exposure image 324.

FIG. 5 is an example of visual representation of a highlight map 500, according to various aspects of the present disclosure. In highlight map 500 light pixels represent areas of high luminance (also referred to as “bright regions” ) and dark pixels represent areas of low luminance (also referred to as “dark regions” ) . Highlight map 500 may represent the same scene represented by short-and-long-exposure motion map 400.

Returning to FIG. 3, motion detector 328 may receive processed short-exposure image 320 from pre-processor 312 and processed short-exposure image 322 from pre-processor 314. Motion detector 328 may compare processed short-exposure image 320 with processed short-exposure image 322 to generate short-exposure motion map 334. Short-exposure motion map 334 may be based on differences between processed short-exposure image 320 and processed short-exposure image 322.

Short-exposure motion map 334 may include a number of indications of differences between pixels of processed short-exposure image 320 and processed short-exposure image 322. For example, short-exposure motion map 334 may include a difference value for each pixel in processed short-exposure image 320, each difference value indicative of a degree of difference between a respective pixel of processed short-exposure image 320 and a corresponding pixel of processed short-exposure image 322. Each of the difference values may correspond to two pixels (e.g., a pixel of processed short-exposure image 320 and a corresponding pixel of processed short-exposure image 322) . As such, each of the difference values of short-exposure motion map 334 may be referred to as a pixel. For descriptive purposes, short-exposure motion map 334 may be visualized (e.g., with black and dark gray representing low degrees of difference and white and light gray representing high degrees of difference) .

FIG. 6 is an example visual representation of a short-exposure motion map 600, according to various aspects of the present disclosure. Short-exposure motion map 600 may be an example of short-exposure motion map 334 of FIG. 3. Short-exposure motion map 600 may represent the same scene represented by short-and-long-exposure motion map 400.

Returning to FIG. 3, because processed short-exposure image 320 and processed short-exposure image 322 may be captured using similar image-capture parameters, processed short-exposure image 320 and processed short-exposure image 322 may exhibit similar noise. Because the noise may be similar (e.g., not different between processed short-exposure image 320 and processed short-exposure image 322) the noise may cancel when generating short-exposure motion map 334. Thus, noise in processed short-exposure image 320 and processed short-exposure image 322 may result in fewer false positives in short-exposure motion map 334 than in short-and-long-exposure motion map 336. With fewer false positives, short-exposure motion map 334 may be more accurate than other motion maps of other HDR techniques (e.g., motion maps based on comparisons between a short-exposure image and a long-exposure image) .

For example, comparing short-and-long-exposure motion map 400 of FIG. 4 to short-exposure motion map 600 of FIG. 6, short-exposure motion map 600 may include fewer false positives than short-and-long-exposure motion map 400. For example, because noise in the regions of the short-exposure images (e.g., processed short-exposure image 320 and processed short-exposure image 322) representing the buildings may be similar, it may cancel and not be represented as differences in short-exposure motion map 600. Further, a hand may move in the scene. The motion of the hand may be visible in short-exposure motion map 600 as the two short-exposure images used to generate short-exposure motion map 600 (e.g., processed short-exposure image 320 and processed short-exposure image 322) may be captured at different times and the hand may be in different positions at the different times.

Returning to FIG. 3, motion detector 326 may receive processed long-exposure image 318 from pre-processor 310 and processed long-exposure image 324 from pre-processor 316. Motion detector 326 may compare processed long-exposure image 318 with processed long-exposure image 324 to generate long-exposure motion map 332. Long-exposure motion map 332 may be based on differences between processed long-exposure image 318 and processed long-exposure image 324.

Long-exposure motion map 332 may include a number of indications of differences between pixels of processed long-exposure image 318 and processed long-exposure image 324. For example, long-exposure motion map 332 may include a difference value for each pixel in processed long-exposure image 318, each difference value indicative of a degree of difference between a respective pixel of processed long-exposure image 318 and a corresponding pixel of processed long-exposure image 324. Each of the difference values may correspond to two pixels (e.g., a pixel of processed long-exposure image 318 and a corresponding pixel of processed long-exposure image 324) . As such, each of the difference values of long-exposure motion map 332 may be referred to as a pixel. For descriptive purposes, long-exposure motion map 332 may be visualized (e.g., with black and dark gray representing low degrees of difference and white and light gray representing high degrees of difference) .

FIG. 7 is an example visual representation of a long-exposure motion map 700, according to various aspects of the present disclosure. Long-exposure motion map 700 may be an example of long-exposure motion map 332 of FIG. 3. Long-exposure motion map 700 may represent the same scene represented by short-and-long-exposure motion map 400.

Returning to FIG. 3, because processed long-exposure image 318 and processed long-exposure image 324 may be captured using similar image-capture parameters, processed long-exposure image 318 and processed long-exposure image 324 may exhibit similar noise. Because the noise may be similar (e.g., not different between processed long-exposure image 318 and processed long-exposure image 324) the noise may cancel when generating long-exposure motion map 332. Thus, noise in processed long-exposure image 318 and processed long-exposure image 324 may result in fewer false positives in long-exposure motion map 332 than in short-and-long-exposure motion map 336. With fewer false positives, long-exposure motion map 332 may be more accurate than other motion maps of other HDR techniques (e.g., motion maps based on comparisons between a short-exposure image and a long-exposure image) .

For example, comparing short-and-long-exposure motion map 400 of FIG. 4 to long-exposure motion map 700 of FIG. 7, long-exposure motion map 700 may include fewer false positives than short-and-long-exposure motion map 400. For example, because noise in the regions of the long-exposure images (e.g., processed long-exposure image 318 and processed long-exposure image 324) representing the buildings may be similar, it may cancel and not be represented as differences in long-exposure motion map 700. Further, a hand may move in the scene. The motion of the hand may be visible in long-exposure motion map 700 as the two long-exposure images used to generate long-exposure motion map 700 (e.g., processed long-exposure image 318 and processed long-exposure image 324) may be captured at different times and the hand may be in different positions at the different times.

Comparing short-exposure motion map 600 of FIG. 6 to long-exposure motion map 700 of FIG. 7, more of the pixels of long-exposure motion map 700 may indicate differences based on the moving hand than the pixels of short-exposure motion map 600. The hand movement may affect more pixels in long-exposure motion map 700 because the hand may move more during the capturing of the long-exposure images used to generate long-exposure motion map 700 (e.g., processed long-exposure image 318 and processed long-exposure image 324) than the hand moves during the capturing of the short-exposure images used to generate short-exposure motion map 600 (e.g., processed short-exposure image 320 and processed short-exposure image 322) .

Map generator 342 may receive long-exposure motion map 332 from motion detector 326 and short-exposure motion map 334 from motion detector 328. Map generator 342 may generate reference motion map 344 based on long-exposure motion map 332 and short-exposure motion map 334. In some instances, map generator 342 may select for reference motion map 344 the maximum difference value for each pixel of long-exposure motion map 332 and short-exposure motion map 334. For example, for each pixel in long-exposure motion map 332 and each corresponding pixel of short-exposure motion map 334, map generator 342 may take greater difference value. For example, expressed using a mathematical function, map generator 342 may generate reference motion map 344 as a per-pixel maximum of long-exposure motion map 332 and short-exposure motion map 334, or max (long-exposure motion map 332, short-exposure motion map 334) .

FIG. 8 is an example visual representation of a reference motion map 800, according to various aspects of the present disclosure. Reference motion map 800 may be an example of reference motion map 344 of FIG. 3. Reference motion map 344 may represent the same scene represented by short-and-long-exposure motion map 400. Further, reference motion map 800 may be a per-pixel maximum of short-exposure motion map 600 of FIG. 6 and long-exposure motion map 700 of FIG. 7.

Returning to FIG. 3, map generator 346 may receive reference motion map 344 from map generator 342, short-and-long-exposure motion map 336 and highlight detector 330 and highlight map 338 from motion detector and highlight detector 330. Map generator 346 may generate fusion map 348 based on reference motion map 344, short-and-long-exposure motion map 336, and highlight map 338. Map generator 346 may generate a final motion map (not illustrated in FIG. 3) based on short-exposure motion map 334 and short-and-long-exposure motion map 336. In some instances, map generator 346 may select for the final motion map the minimum difference value for each pixel of reference motion map 344 and short-and-long- exposure motion map 336. For example, for each pixel in reference motion map 344 and each corresponding pixel of short-and-long-exposure motion map 336, map generator 346 may take lesser difference value. For example, expressed using a mathematical function, map generator 346 may generate the final motion map as a per-pixel minimum of reference motion map 344 and short-and-long-exposure motion map 336, or min (reference motion map 344, short-and-long-exposure motion map 336) .

FIG. 9 is an example visual representation of a final motion map 900, according to various aspects of the present disclosure. Final motion map 900 may represent the same scene represented by short-and-long-exposure motion map 400. Further, final motion map 900 may be a per-pixel minimum of reference motion map 800 of FIG. 8 and short-and-long-exposure motion map 400 of FIG. 4.

Returning to FIG. 3, map generator 346 may generate fusion map 348 based on the final motion map (not illustrated in FIG. 3) and highlight map 338. Map generator 346 may combine the final motion map with highlight map 338 according to any suitable means for combining pixel maps, such as, using a max function. The final motion map and/or highlight map 338 may be scaled before they are combined.

FIG. 10 is an example visual representation of a fusion map 1000, according to various aspects of the present disclosure. Fusion map 1000 may represent the same scene represented by short-and-long-exposure motion map 400. Further, fusion map 1000 may be based on final motion map 900 of FIG. 9 and highlight map 500 of FIG. 5.

Returning to FIG. 3, combiner 350 may use fusion map 348 to determine from which of processed short-exposure image 322 and processed long-exposure image 324 to select pixels to include in composite image 352. Images 340 may include processed short-exposure image 322 and processed long-exposure image 324. Combiner 350 may receive images 340 from motion detector and highlight detector 330. Alternatively, combiner 350 may receive processed short-exposure image 322 from pre-processor 314 and processed long-exposure image 324 from pre-processor 316. In any case, combiner 350 may select pixels from processed short-exposure image 322 and pixels from processed long-exposure image 324 to generate composite image 352 based on fusion map 348.

Using fusion map 1000 of FIG. 10 as an example, combiner 350 may select pixels from processed short-exposure image 322 that correspond to light pixels of fusion map 1000 (e.g., pixels of fusion map 1000 that have a value of greater than 128 when each pixel of fusion map 1000 is represented by an 8-bit value) . For example, based on highlight map 338, the sky region of the scene may be bright. Combiner 350 may select pixels from processed short-exposure image 322 representative of the bright sky region for inclusion in composite image 352. Selecting pixels from processed short-exposure image 322 to represent bright regions of the scene may preserve details of the bright region captured by processed short-exposure image 322.

In some aspects, one or more elements and/or corresponding operations of system 300 may be omitted. For example, in some aspects, short-and-long-exposure motion map 336 may be omitted and map generator 346 may generate fusion map 348 based on reference motion map 344 and highlight map 338. In such aspects, combiner 350 may generate composite image 352 based on long-exposure motion map 332, short-exposure motion map 334, and highlight map 338. As another example, in some aspects, motion detector 326 and long-exposure motion map 332 may be omitted and map generator 346 may generate fusion map 348 based on short-exposure motion map 334, short-and-long-exposure motion map 336, and highlight map 338. In such aspects, combiner 350 may generate composite image 352 based on short-exposure motion map 334, short-and-long-exposure motion map 336, and highlight map 338. As another example, in some aspects, short-and-long-exposure motion map 336, motion detector 326, and long-exposure motion map 332 may be omitted and map generator 346 may generate fusion map 348 based on based on short-exposure motion map 334 and highlight map 338. In such aspects, combiner 350 may generate composite image 352 based on short-exposure motion map 334 and highlight map 338. As another example, in some aspects, motion detector 328 and short-exposure motion map 334 may be omitted and map generator 346 may generate fusion map 348 based on based on long-exposure motion map 332, short-and-long-exposure motion map 336, and highlight map 338. In such aspects, combiner 350 may generate composite image 352 based on long-exposure motion map 332, short-and-long-exposure motion map 336, and highlight map 338. As another example, in some aspects, short-and-long-exposure motion map 336, motion detector 328, and short-exposure motion map 334 may be omitted and map generator 346 may generate fusion map 348 based on based on long-exposure motion map 332 and highlight map 338. In such aspects, combiner 350 may generate composite image 352 based on long-exposure motion map 332 and highlight map 338.

FIG. 11 is a flow diagram illustrating a process 1100 for generating a composite image, in accordance with aspects of the present disclosure. One or more operations of process 1100 may be performed by a computing device (or apparatus) or a component (e.g., a chipset, codec, etc. ) of the computing device. The computing device may be a mobile device (e.g., a mobile phone) , a network-connected wearable such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or augmented reality (AR) device, a vehicle or component or system of a vehicle, or other type of computing device. The one or more operations of process 1100 may be implemented as software components that are executed and run on one or more processors.

At block 1102, a computing device (or one or more components thereof) may obtain a first short-exposure image associated with a first exposure duration. For example, system 300 of FIG. 3 may obtain short-exposure image 304 of FIG. 3.

At block 1104, the computing device (or one or more components thereof) may obtain a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration. For example, system 300 may obtain long-exposure image 302 of FIG. 3.

At block 1106, the computing device (or one or more components thereof) may obtain a second short-exposure image associated with the first exposure duration. For example, system 300 of FIG. 3 may obtain short-exposure image 306 of FIG. 3.

At block 1108, the computing device (or one or more components thereof) may obtain a second long-exposure image associated with the second exposure duration. For example, system 300 may obtain long-exposure image 308 of FIG. 3.

In some aspects, the computing device (or one or more components thereof) may perform a black-level-deduction operation on the second short-exposure image and the second long-exposure image. For example, pre-processor 314 and pre-processor 316 may perform a black-level deduction operation on short-exposure image 306 and long-exposure image 308 respectively. In some aspects, the computing device (or one or more components thereof) may compensate the second short-exposure image based on an exposure ratio between the second short-exposure image and the second long-exposure image. For example, pre-processor 314 may compensate short-exposure image 306 based on an exposure ratio between short-exposure image 306 and long-exposure image 308. In some aspects, the computing device (or one or more components thereof) may apply a low-pass filter to the second short-exposure image. For example, pre-processor 314 may low-pass filter short-exposure image 306.

At block 1110, the computing device (or one or more components thereof) may compare the first short-exposure image with the second short-exposure image. For example, system 300, using motion detector 328, may compare short-exposure image 304 with short-exposure image 306.

In some aspects, the computing device (or one or more components thereof) may, to compare the first short-exposure image with the second short-exposure image, determine a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first short-exposure image and a corresponding pixel of the second short-exposure image. For example, motion detector 328 may determine a difference value for each pixel of short-exposure image 304 and short-exposure image 306. In some aspects, the computing device (or one or more components thereof) may generate a short-exposure motion map at least in part by thresholding the plurality of difference values. For example, motion detector 328 may threshold the difference values to generate short-exposure motion map 334. In some aspects, to threshold the plurality of difference values, the at least one processor may compare each of the plurality of difference values to one or more threshold values. Difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a high value. Difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a low value. Difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the short-exposure motion map with respective values between the high value and the low value.

In some aspects, the computing device (or one or more components thereof) may, to compare the first long-exposure image with the second long-exposure image, determine a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first long-exposure image and a corresponding pixel of the second long-exposure image. For example, motion detector 326 may determine a difference value for each pixel of long-exposure image 302 and long-exposure image 308. In some aspects, the computing device (or one or more components thereof) may generate a long-exposure motion map at least in part by thresholding the plurality of difference values. For example, motion detector 326 may threshold the difference values to generate long-exposure motion map 332. In some aspects, to threshold the plurality of difference values, the computing device (or one or more components thereof) may compare each of the plurality of difference values to one or more threshold values. Difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a high value. Difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a low value. Difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the long-exposure motion map with respective values between the high value and the low value

At block 1112, the computing device (or one or more components thereof) may compare the first long-exposure image with the second long-exposure image. For example, system 300, using motion detector 326, may compare long-exposure image 302 with long-exposure image 308.

At block 1114, the computing device (or one or more components thereof) may determine a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image. For example, system 300, using map generator 342, may generate reference motion map 344 based on short-exposure motion map 334 (which is based on the comparison of short-exposure image 304 and short-exposure image 306) and long-exposure motion map 332 (which is based on the comparison of long-exposure image 302 and long-exposure image 308) . Further, system 300, using map generator 346, may generate fusion map 348 based on reference motion map 344.

In some aspects, the computing device (or one or more components thereof) may generate a short-exposure motion map based on comparing between the first short-exposure image and the second short-exposure image and generate a long-exposure motion map based on comparing between the first long-exposure image with the second long-exposure image. The motion map may be determined based on the short-exposure motion map and the long-exposure motion map. For example, system 300, using motion detector 328, may generate short-exposure motion map 334 based on comparing short-exposure image 304 with short-exposure image 306. Further, system 300, using motion detector 326, may generate long-exposure motion map 332 based on comparing long-exposure image 302 with long-exposure image 308. Further, system 300, using map generator 346, may generate fusion map 348 based on processed short-exposure image 322 and short-exposure motion map 334.

In some aspects, the computing device (or one or more components thereof) may generate a reference motion map based on the short-exposure motion map and the long-exposure motion map; wherein the motion map is determined based on the reference motion map. For example, system 300, using map generator 342, may generate reference motion map 344 based on long-exposure motion map 332 and short-exposure motion map 334. In some aspects, to generate the reference motion map, the computing device (or one or more components thereof) may select, for each pixel of the reference motion map, a greater of a corresponding value of the short-exposure motion map or a corresponding value of the long-exposure motion map. For example, to generate reference motion map 344, map generator 342 may take for each pixel of long-exposure motion map 332 and short-exposure motion map 334 a maximum value.

In some aspects, the computing device (or one or more components thereof) may compare the second short-exposure image with the second long-exposure image; wherein the motion map is further determined based on comparing between the second short-exposure image and the second long-exposure image. In some aspects, the computing device (or one or more components thereof) may generate a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image. The motion map may be determined further based on the short-and-long-exposure motion map. For example, system 300, using highlight detector 330, may determine short-and-long-exposure motion map 336 based on a comparison between short-exposure image 306 and long-exposure image 308. Further, system 300, using map generator 346 may generate fusion map 348 based, in part, on short-and-long-exposure motion map 336.

In some aspects, the computing device (or one or more components thereof) may generate a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image and generate a reference motion map based on the short-exposure motion map and the long-exposure motion map. The motion map may be determined by selecting, for each pixel of the motion map, a lesser of a corresponding value of the short-and-long-exposure motion map or a corresponding value of the reference motion map. For example, system 300, using highlight detector 330, may determine short-and-long-exposure motion map 336 based on a comparison between short-exposure image 306 and long-exposure image 308. Further, system 300, using map generator 342, may generate reference motion map 344 based on long-exposure motion map 332 and short-exposure motion map 334. Further, system 300, using map generator 346 may determine fusion map 348 based on reference motion map 344 and short-and-long-exposure motion map 336. For example, to generate fusion map 348, map generator 346 may take for each pixel of reference motion map 344 and short-and-long-exposure motion map 336 a minimum value.

In some aspects, the computing device (or one or more components thereof) may determine a highlight map based on one or both of the second short-exposure image or the second long-exposure image. The composite image may be generated further based on the highlight map. In some aspects, the computing device (or one or more components thereof) may determine a highlight map based on one or both of the second short-exposure image or the second long-exposure image and generate a fusion map based on the motion map and the highlight map. The composite image may be generated further based on the fusion map. For example, system 300, using highlight detector 330, may determine highlight map 338. Map generator 346 may generate fusion map 348 based, in part, on highlight map 338 and combiner 350 may generate composite image 352 based, in part, on fusion map 348.

At block 1116, the computing device (or one or more components thereof) may generate a composite image based on the second short-exposure image, the second long-exposure image, and the motion map. For example, system 300, using combiner 350, may generate composite image 352 based on fusion map 348.

In some aspects, to generate the composite image, the computing device (or one or more components thereof) may select, for inclusion in the composite image based on the motion map, first pixels from the second short-exposure image and second pixels from the second long-exposure image. For example, to generate composite image 352, system 300, using combiner 350, may select pixels from short-exposure image 306 and pixels from long-exposure image 308 for inclusion in composite image 352.

In some aspects, to generate the composite image, the computing device (or one or more components thereof) may blend first pixels from the second short-exposure image and second pixels from the second long-exposure image based on the motion map. For example, to generate composite image 352, system 300, using combiner 350, may blend pixels from short-exposure image 306 and pixels from long-exposure image 308 to generate pixels for composite image 352.

In some examples, the methods described herein (e.g., process 1100 of FIG. 11 and/or other methods described herein) can be performed, in whole or in part, by a computing device or apparatus. In one example, one or more of the methods can be performed by image-processing system 100 of FIG. 1, image-processing device 104 of FIG. 1, system 300 of FIG. 3 (or one or more elements thereof) , or by another system or device. In another example, one or more of the methods can be performed, in whole or in part, by the computing-device architecture 1200 shown in FIG. 12. For instance, a computing device with the computing-device architecture 1200 shown in FIG. 12 can include, or be included in, the components of the image-processing system 100 of FIG. 1, image-processing device 104 of FIG. 1, system 300 of FIG. 3 (or one or more elements thereof) and can implement the operations of process 1100 and/or other process described herein. The computing device can include any suitable device, such as a vehicle or a computing device of a vehicle, a mobile device (e.g., a mobile phone) , a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device) , a server computer, a robotic device, a television, and/or any other computing device with the resource capabilities to perform the processes described herein, including process 1100 and/or other process described herein. In some cases, the computing device or apparatus can include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component (s) that are configured to carry out the steps of processes described herein. In some examples, the computing device can include a display, a network interface configured to communicate and/or receive the data, any combination thereof, and/or other component (s) . The network interface can be configured to communicate and/or receive Internet Protocol (IP) based data or other type of data.

The components of the computing device can be implemented in circuitry. For example, the components can include and/or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs) , digital signal processors (DSPs) , central processing units (CPUs) , and/or other suitable electronic circuits) , and/or can include and/or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

Process 1100 and/or other process described herein are illustrated as logical flow diagrams, the operation of which represents a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.

Additionally, process 1100 and/or other process described herein can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

FIG. 12 illustrates an example computing-device architecture 1200 of an example computing device which can implement the various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device) , a personal computer, a laptop computer, a video server, a vehicle (or computing device of a vehicle) , or other device. For example, the computing-device architecture 1200 may include, implement, or be included in any or all of image-processing system 100 of FIG. 1, image-processing device 104 of FIG. 1, system 300 of FIG. 3 (or one or more elements thereof) .

The components of computing-device architecture 1200 are shown in electrical communication with each other using connection 1212, such as a bus. The example computing-device architecture 1200 includes a processing unit (CPU or processor) 1202 and computing device connection 1212 that couples various computing device components including computing device memory 1210, such as read only memory (ROM) 1208 and random-access memory (RAM) 1206, to processor 1202.

Computing-device architecture 1200 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1202. Computing-device architecture 1200 can copy data from memory 1210 and/or the storage device 1214 to cache 1204 for quick access by processor 1202. In this way, the cache can provide a performance boost that avoids processor 1202 delays while waiting for data. These and other modules can control or be configured to control processor 1202 to perform various actions. Other computing device memory 1210 may be available for use as well. Memory 1210 can include multiple different types of memory with different performance characteristics. Processor 1202 can include any general-purpose processor and a hardware or software service, such as service 1 1216, service 2 1218, and service 3 1220 stored in storage device 1214, configured to control processor 1202 as well as a special-purpose processor where software instructions are incorporated into the processor design. Processor 1202 may be a self-contained system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing-device architecture 1200, input device 1222 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. Output device 1224 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display, projector, television, speaker device, etc. In some instances, multimodal computing devices can enable a user to provide multiple types of input to communicate with computing-device architecture 1200. Communication interface 1226 can generally govern and manage the user input and computing device output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device 1214 is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random-access memories (RAMs) 1206, read only memory (ROM) 1208, and hybrids thereof. Storage device 1214 can include services 1216, 1218, and 1220 for controlling processor 1202. Other hardware or software modules are contemplated. Storage device 1214 can be connected to the computing device connection 1212. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1202, connection 1212, output device 1224, and so forth, to carry out the function.

The term “substantially, ” in reference to a given parameter, property, or condition, may refer to a degree that one of ordinary skill in the art would understand that the given parameter, property, or condition is met with a small degree of variance, such as, for example, within acceptable manufacturing tolerances. By way of example, depending on the particular parameter, property, or condition that is substantially met, the parameter, property, or condition may be at least 90%met, at least 95%met, or even at least 99%met.

Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to specific devices.

The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system and so on) . As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term “device” to describe various aspects of this disclosure, the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. While the below description and examples use the term “system” to describe various aspects of this disclosure, the term “system” is not limited to a specific configuration, type, or number of objects.

Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks including devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and/or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.

Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general-purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction (s) and/or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and/or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD) , flash memory, magnetic or optical disks, USB devices provided with non-volatile memory, networked storage devices, any suitable combination thereof, among others. A computer-readable medium may have stored thereon code and/or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

In some aspects the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor (s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

In the foregoing description, aspects of the application are described with reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.

One of ordinary skill will appreciate that the less than ( “<” ) and greater than ( “>” ) symbols or terminology used herein can be replaced with less than or equal to ( “≤” ) and greater than or equal to ( “≥” ) symbols, respectively, without departing from the scope of this description.

Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.

The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and/or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and/or other suitable communication interface) either directly or indirectly.

Claim language or other language reciting “at least one of” a set and/or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of” a set and/or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general-purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random-access memory (RAM) such as synchronous dynamic random-access memory (SDRAM) , read-only memory (ROM) , non-volatile random-access memory (NVRAM) , electrically erasable programmable read-only memory (EEPROM) , FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as propagated signals or waves.

The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs) , general-purpose microprocessors, an application specific integrated circuits (ASICs) , field programmable logic arrays (FPGAs) , or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration) . Accordingly, the term “processor, ” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

Illustrative aspects of the disclosure include:

Aspect 1. An apparatus for generating a composite image, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain a first short-exposure image associated with a first exposure duration; obtain a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; obtain a second short-exposure image associated with the first exposure duration; obtain a second long-exposure image associated with the second exposure duration; compare the first short-exposure image with the second short-exposure image; compare the first long-exposure image with the second long-exposure image; determine a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and generate a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

Aspect 2. The apparatus of aspect 1, wherein the at least one processor is further configured to: generate a short-exposure motion map based on comparing between the first short-exposure image and the second short-exposure image; and generate a long-exposure motion map based on comparing between the first long-exposure image with the second long-exposure image; wherein the motion map is determined based on the short-exposure motion map and the long-exposure motion map.

Aspect 3. The apparatus of aspect 2, wherein the at least one processor is further configured to generate a reference motion map based on the short-exposure motion map and the long-exposure motion map; wherein the motion map is determined based on the reference motion map.

Aspect 4. The apparatus of aspect 3, wherein to generate the reference motion map, the at least one processor is further configured to select, for each pixel of the reference motion map, a greater of a corresponding value of the short-exposure motion map or a corresponding value of the long-exposure motion map.

Aspect 5. The apparatus of any one of aspects 2 to 4, wherein: the at least one processor is further configured to generate a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image; and the motion map is determined further based on the short-and-long-exposure motion map.

Aspect 6. The apparatus of any one of aspects 2 to 5, wherein the at least one processor is further configured to: generate a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image; and generate a reference motion map based on the short-exposure motion map and the long-exposure motion map; wherein the motion map is determined by selecting, for each pixel of the motion map, a lesser of a corresponding value of the short-and-long-exposure motion map or a corresponding value of the reference motion map.

Aspect 7. The apparatus of aspect 6, wherein the at least one processor is further configured to: determine a highlight map based on one or both of the second short-exposure image or the second long-exposure image; and generate a fusion map based on the motion map and the highlight map; wherein the composite image is generated further based on the fusion map.

Aspect 8. The apparatus of any one of aspects 1 to 7, wherein to compare the first short-exposure image with the second short-exposure image, the at least one processor is further configured to determine a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first short-exposure image and a corresponding pixel of the second short-exposure image.

Aspect 9. The apparatus of aspect 8, wherein the at least one processor is further configured to generate a short-exposure motion map at least in part by thresholding the plurality of difference values.

Aspect 10. The apparatus of aspect 9, wherein: to threshold the plurality of difference values, the at least one processor is further configured to compare each of the plurality of difference values to one or more threshold values; difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a high value; difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a low value; and difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the short-exposure motion map with respective values between the high value and the low value.

Aspect 11. The apparatus of any one of aspects 1 to 10, wherein to compare the first long-exposure image with the second long-exposure image, the at least one processor is further configured to determine a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first long-exposure image and a corresponding pixel of the second long-exposure image.

Aspect 12. The apparatus of aspect 11, wherein the at least one processor is further configured to generate a long-exposure motion map at least in part by thresholding the plurality of difference values.

Aspect 13. The apparatus of aspect 12, wherein: to threshold the plurality of difference values, wherein the at least one processor is further configured to compare each of the plurality of difference values to one or more threshold values; difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a high value; difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a low value; and difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the long-exposure motion map with respective values between the high value and the low value.

Aspect 14. The apparatus of any one of aspects 1 to 13, wherein the at least one processor is further configured to perform a black-level-deduction operation on the second short-exposure image and the second long-exposure image.

Aspect 15. The apparatus of any one of aspects 1 to 14, wherein the at least one processor is further configured to compensate the second short-exposure image based on an exposure ratio between the second short-exposure image and the second long-exposure image.

Aspect 16. The apparatus of any one of aspects 1 to 15, wherein the at least one processor is further configured to apply a low-pass filter to the second short-exposure image.

Aspect 17. The apparatus of any one of aspects 1 to 16, wherein the at least one processor is further configured to compare the second short-exposure image with the second long-exposure image; wherein the motion map is further determined based on comparing between the second short-exposure image and the second long-exposure image.

Aspect 18. The apparatus of any one of aspects 1 to 17, wherein the at least one processor is further configured to determine a highlight map based on one or both of the second short-exposure image or the second long-exposure image; wherein the composite image is generated further based on the highlight map.

Aspect 19. The apparatus of any one of aspects 1 to 18, wherein to generate the composite image, the at least one processor is further configured to select, for inclusion in the composite image based on the motion map, first pixels from the second short-exposure image and second pixels from the second long-exposure image.

Aspect 20. The apparatus of any one of aspects 1 to 19, wherein to generate the composite image, the at least one processor is further configured to blend first pixels from the second short-exposure image and second pixels from the second long-exposure image based on the motion map.

Aspect 21. A method for generating a composite image, the method comprising: obtaining a first short-exposure image associated with a first exposure duration; obtaining a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration; obtaining a second short-exposure image associated with the first exposure duration; obtaining a second long-exposure image associated with the second exposure duration; comparing the first short-exposure image with the second short-exposure image; comparing the first long-exposure image with the second long-exposure image; determining a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and generating a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.

Aspect 22. The method of aspect 21, further comprising: generating a short-exposure motion map based on comparing between the first short-exposure image and the second short-exposure image; and generating a long-exposure motion map based on comparing between the first long-exposure image with the second long-exposure image; wherein the motion map is determined based on the short-exposure motion map and the long-exposure motion map.

Aspect 23. The method of aspect 22, further comprising generating a reference motion map based on the short-exposure motion map and the long-exposure motion map; wherein the motion map is determined based on the reference motion map.

Aspect 24. The method of aspect 23, wherein generating the reference motion map comprises selecting, for each pixel of the reference motion map, a greater of a corresponding value of the short-exposure motion map or a corresponding value of the long-exposure motion map.

Aspect 25. The method of any one of aspects 22 to 24, further comprising generating a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image, wherein the motion map is determined further based on the short-and-long-exposure motion map.

Aspect 26. The method of any one of aspects 22 to 25, further comprising: generating a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image; and generating a reference motion map based on the short-exposure motion map and the long-exposure motion map; wherein the motion map is determined by selecting, for each pixel of the motion map, a lesser of a corresponding value of the short-and-long-exposure motion map or a corresponding value of the reference motion map.

Aspect 27. The method of aspect 26, further comprising: determining a highlight map based on one or both of the second short-exposure image or the second long-exposure image; and generating a fusion map based on the motion map and the highlight map; wherein the composite image is generated further based on the fusion map.

Aspect 28. The method of any one of aspects 21 to 27, wherein comparing the first short-exposure image with the second short-exposure image comprises determining a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first short-exposure image and a corresponding pixel of the second short-exposure image.

Aspect 29. The method of aspect 28, further comprising generating a short-exposure motion map at least in part by thresholding the plurality of difference values.

Aspect 30. The method of aspect 29, wherein: thresholding the plurality of difference values comprises comparing each of the plurality of difference values to one or more threshold values; difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a high value; difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a low value; and difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the short-exposure motion map with respective values between the high value and the low value.

Aspect 31. The method of any one of aspects 21 to 30, wherein comparing the first long-exposure image with the second long-exposure image comprises determining a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first long-exposure image and a corresponding pixel of the second long-exposure image.

Aspect 32. The method of aspect 31, further comprising generating a long-exposure motion map at least in part by thresholding the plurality of difference values.

Aspect 33. The method of aspect 32, wherein: thresholding the plurality of difference values comprises comparing each of the plurality of difference values to one or more threshold values; difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a high value; difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a low value; and difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the long-exposure motion map with respective values between the high value and the low value.

Aspect 34. The method of any one of aspects 21 to 33, further comprising performing a black-level-deduction operation on the second short-exposure image and the second long-exposure image.

Aspect 35. The method of any one of aspects 21 to 34, further comprising compensating the second short-exposure image based on an exposure ratio between the second short-exposure image and the second long-exposure image.

Aspect 36. The method of any one of aspects 21 to 35, further comprising applying a low-pass filter to the second short-exposure image.

Aspect 37. The method of any one of aspects 21 to 36, further comprising comparing the second short-exposure image with the second long-exposure image; wherein the motion map is further determined based on comparing between the second short-exposure image and the second long-exposure image.

Aspect 38. The method of any one of aspects 21 to 37, further comprising determining a highlight map based on one or both of the second short-exposure image or the second long-exposure image; wherein the composite image is generated further based on the highlight map.

Aspect 39. The method of any one of aspects 21 to 38, wherein generating the composite image comprises selecting, for inclusion in the composite image based on the motion map, first pixels from the second short-exposure image and second pixels from the second long-exposure image.

Aspect 40. The method of any one of aspects 21 to 39, wherein generating the composite image comprises blending first pixels from the second short-exposure image and second pixels from the second long-exposure image based on the motion map.

Aspect 41. A method for generating a composite image, the method comprising: obtaining a first short-exposure image associated with a first exposure duration; obtaining a second short-exposure image associated with the first exposure duration; obtaining a long-exposure image associated with the second exposure duration, wherein the second exposure duration is greater than the first exposure duration; comparing the first short-exposure image with the second short-exposure image; determining a motion map based on comparing the first short-exposure image and the second short-exposure image; and generating a composite image based on the second short-exposure image, the long-exposure image, and the motion map.

Aspect 42. A method for generating a composite image, the method comprising: obtaining a short-exposure image associated with a first exposure duration; obtaining a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is greater than the first exposure duration; obtaining a second long-exposure image associated with the second exposure duration; comparing the first long-exposure image with the second long-exposure image; determining a motion map based on comparing the first long-exposure image and the second long-exposure image; and generating a composite image based on the short-exposure image, the second long-exposure image, and the motion map.

Aspect 43. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations according to any of aspects 21 to 42.

Aspect 44. An apparatus for providing virtual content for display, the apparatus comprising one or more means for perform operations according to any of aspects 21 to 42.

Claims

An apparatus for generating a composite image, the apparatus comprising:

at least one memory; and

at least one processor coupled to the at least one memory and configured to:

obtain a first short-exposure image associated with a first exposure duration;

obtain a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration;

obtain a second short-exposure image associated with the first exposure duration;

obtain a second long-exposure image associated with the second exposure duration;

compare the first short-exposure image with the second short-exposure image;

compare the first long-exposure image with the second long-exposure image;

determine a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and

generate a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.
The apparatus of claim 1, wherein the at least one processor is further configured to:

generate a short-exposure motion map based on comparing between the first short-exposure image and the second short-exposure image; and

generate a long-exposure motion map based on comparing between the first long-exposure image with the second long-exposure image;

wherein the motion map is determined based on the short-exposure motion map and the long-exposure motion map.
The apparatus of claim 2, wherein the at least one processor is further configured to generate a reference motion map based on the short-exposure motion map and the long- exposure motion map; wherein the motion map is determined based on the reference motion map.
The apparatus of claim 3, wherein to generate the reference motion map, the at least one processor is further configured to select, for each pixel of the reference motion map, a greater of a corresponding value of the short-exposure motion map or a corresponding value of the long-exposure motion map.
The apparatus of claim 2, wherein:

the at least one processor is further configured to generate a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image; and

the motion map is determined further based on the short-and-long-exposure motion map.
The apparatus of claim 2, wherein the at least one processor is further configured to:

generate a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image; and

generate a reference motion map based on the short-exposure motion map and the long-exposure motion map;

wherein the motion map is determined by selecting, for each pixel of the motion map, a lesser of a corresponding value of the short-and-long-exposure motion map or a corresponding value of the reference motion map.
The apparatus of claim 6, wherein the at least one processor is further configured to:

determine a highlight map based on one or both of the second short-exposure image or the second long-exposure image; and

generate a fusion map based on the motion map and the highlight map;

wherein the composite image is generated further based on the fusion map.
The apparatus of claim 1, wherein to compare the first short-exposure image with the second short-exposure image, the at least one processor is further configured to determine a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first short-exposure image and a corresponding pixel of the second short-exposure image.
The apparatus of claim 8, wherein the at least one processor is further configured to generate a short-exposure motion map at least in part by thresholding the plurality of difference values.
The apparatus of claim 9, wherein:

to threshold the plurality of difference values, the at least one processor is further configured to compare each of the plurality of difference values to one or more threshold values;

difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a high value;

difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a low value; and

difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the short-exposure motion map with respective values between the high value and the low value.
The apparatus of claim 1, wherein to compare the first long-exposure image with the second long-exposure image, the at least one processor is further configured to determine a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first long-exposure image and a corresponding pixel of the second long-exposure image.
The apparatus of claim 11, wherein the at least one processor is further configured to generate a long-exposure motion map at least in part by thresholding the plurality of difference values.
The apparatus of claim 12, wherein:

to threshold the plurality of difference values, wherein the at least one processor is further configured to compare each of the plurality of difference values to one or more threshold values;

difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a high value;

difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the long-exposure motion map with a low value; and

difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the long-exposure motion map with respective values between the high value and the low value.
The apparatus of claim 1, wherein the at least one processor is further configured to perform a black-level-deduction operation on the second short-exposure image and the second long-exposure image.
The apparatus of claim 1, wherein the at least one processor is further configured to compensate the second short-exposure image based on an exposure ratio between the second short-exposure image and the second long-exposure image.
The apparatus of claim 1, wherein the at least one processor is further configured to apply a low-pass filter to the second short-exposure image.
The apparatus of claim 1, wherein the at least one processor is further configured to compare the second short-exposure image with the second long-exposure image; wherein the motion map is further determined based on comparing between the second short-exposure image and the second long-exposure image.
The apparatus of claim 1, wherein the at least one processor is further configured to determine a highlight map based on one or both of the second short-exposure image or the second long-exposure image; wherein the composite image is generated further based on the highlight map.
The apparatus of claim 1, wherein to generate the composite image, the at least one processor is further configured to select, for inclusion in the composite image based on the motion map, first pixels from the second short-exposure image and second pixels from the second long-exposure image.
The apparatus of claim 1, wherein to generate the composite image, the at least one processor is further configured to blend first pixels from the second short-exposure image and second pixels from the second long-exposure image based on the motion map.
A method for generating a composite image, the method comprising:

obtaining a first short-exposure image associated with a first exposure duration;

obtaining a first long-exposure image associated with a second exposure duration, wherein the second exposure duration is longer than the first exposure duration;

obtaining a second short-exposure image associated with the first exposure duration;

obtaining a second long-exposure image associated with the second exposure duration;

comparing the first short-exposure image with the second short-exposure image;

comparing the first long-exposure image with the second long-exposure image;

determining a motion map based on comparing the first short-exposure image and the second short-exposure image and comparing the first long-exposure image and the second long-exposure image; and

generating a composite image based on the second short-exposure image, the second long-exposure image, and the motion map.
The method of claim 21, further comprising:

generating a short-exposure motion map based on comparing between the first short-exposure image and the second short-exposure image; and

generating a long-exposure motion map based on comparing between the first long-exposure image with the second long-exposure image;

wherein the motion map is determined based on the short-exposure motion map and the long-exposure motion map.
The method of claim 22, further comprising generating a reference motion map based on the short-exposure motion map and the long-exposure motion map; wherein the motion map is determined based on the reference motion map.
The method of claim 23, wherein generating the reference motion map comprises selecting, for each pixel of the reference motion map, a greater of a corresponding value of the short-exposure motion map or a corresponding value of the long-exposure motion map.
The method of claim 22, further comprising generating a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image, wherein the motion map is determined further based on the short-and-long-exposure motion map.
The method of claim 22, further comprising:

generating a short-and-long-exposure motion map based on comparing between the second short-exposure image and the second long-exposure image; and

generating a reference motion map based on the short-exposure motion map and the long-exposure motion map;

wherein the motion map is determined by selecting, for each pixel of the motion map, a lesser of a corresponding value of the short-and-long-exposure motion map or a corresponding value of the reference motion map.
The method of claim 26, further comprising:

determining a highlight map based on one or both of the second short-exposure image or the second long-exposure image; and

generating a fusion map based on the motion map and the highlight map;

wherein the composite image is generated further based on the fusion map.
The method of claim 21, wherein comparing the first short-exposure image with the second short-exposure image comprises determining a plurality of difference values, each difference value of the plurality of difference values being indicative of a respective difference between a respective pixel of the first short-exposure image and a corresponding pixel of the second short-exposure image.
The method of claim 28, further comprising generating a short-exposure motion map at least in part by thresholding the plurality of difference values.
The method of claim 29, wherein:

thresholding the plurality of difference values comprises comparing each of the plurality of difference values to one or more threshold values;

difference values of the plurality of difference values that are greater than a high threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a high value;

difference values of the plurality of difference values that are less than a low threshold value of the one or more threshold values result in corresponding pixels of the short-exposure motion map with a low value; and

difference values of the plurality of difference values that are between the high threshold value and the low threshold value result in corresponding pixels of the short-exposure motion map with respective values between the high value and the low value.