[go: up one dir, main page]

WO2022194352A1 - Appareil et procédé de correction de corrélation d'images - Google Patents

Appareil et procédé de correction de corrélation d'images Download PDF

Info

Publication number
WO2022194352A1
WO2022194352A1 PCT/EP2021/056657 EP2021056657W WO2022194352A1 WO 2022194352 A1 WO2022194352 A1 WO 2022194352A1 EP 2021056657 W EP2021056657 W EP 2021056657W WO 2022194352 A1 WO2022194352 A1 WO 2022194352A1
Authority
WO
WIPO (PCT)
Prior art keywords
correlation
image
processing apparatus
input image
image processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2021/056657
Other languages
English (en)
Inventor
Hyun Jun JUNG
Yannick VERDIE
Nikolas BRASCH
Benjamin BUSAM
Ales LEONARDIS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Technische Universitaet Muenchen
Original Assignee
Huawei Technologies Co Ltd
Technische Universitaet Muenchen
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd, Technische Universitaet Muenchen filed Critical Huawei Technologies Co Ltd
Priority to PCT/EP2021/056657 priority Critical patent/WO2022194352A1/fr
Publication of WO2022194352A1 publication Critical patent/WO2022194352A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/4808—Evaluating distance, position or velocity data
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/4865—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak
    • G01S7/4866—Time delay measurement, e.g. time-of-flight measurement, time of arrival measurement or determining the exact position of a peak by fitting a model or function to the received signal
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/483—Details of pulse systems
    • G01S7/486—Receivers
    • G01S7/487—Extracting wanted echo signals, e.g. pulse detection
    • G01S7/4876—Extracting wanted echo signals, e.g. pulse detection by removing unwanted signals
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G01S17/32—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated
    • G01S17/36—Systems determining position data of a target for measuring distance only using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated with phase comparison between the received signal and the contemporaneously transmitted signal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation

Definitions

  • the present invention relates to image formation in digital photography, for example to estimating depth for an image using correlation images obtained from Time-of-Flight data.
  • the returned signal r(t) n signal d(t) with control phase T can be input into the formula, and the sinus function of both phase shift and controlled phase can be obtained:
  • the 1-ToF camera _ _ _ _ _ _ _ _ _ , _ _ the resulting four images are called correlation images. By nature, they have a fixed relationship. As correlation images are sinus functions of , the arc-tangent formula can be used to extract Y . Then the distance can be obtained by converting Y using the speed of light c.
  • the returned signal may not necessarily contain only one reflected signal, as shown in Figure 1 (a).
  • the returned signal can be formulated as:
  • the correlation signals become the original returned signal combined with a linear combination of sinus functions with different phase, which can be treated as noise.
  • Sparsity and ambient noise can also be sources of error.
  • any factor causing scattering of the IR signal can degrade the quality of the raw correlation images, such as the surface normal or the material of the objects in the scene.
  • the amplitude of the returned signal decreases and the depth error increases as the surface normal of the object deviates from the optical axis of the camera and when the object's material is reflective, due to the scattering effect.
  • the signals become highly noisy, as shown in Figure 1 (c) for the correlation images C1-C4.
  • MPI and MA removal is a commonly known approach for addressing the l-ToF depth error.
  • the approach described in US 8786678 tries to detect motion artefacts by enforcing continuity constraints in the correlation images, while Lee et al., ‘Motion blur-free time-of-flight range sensor’, Sensors, Cameras and Systems for Industrial and Scientific Applications XIII, vol. 8298, p. 82980U, International Society for Optics and Photonics, 2012, focuses on the physical relationship in the correlation to detect the motion artefact. Both works use information from the correlation to detect the error. However, they use the information to generate a mask to remove areas, rather than improving the affected area. Other works focus on removing the MPI error, but require a special l-ToF camera with multiple frequencies, or require a specific assumption with computationally heavy optimization.
  • an image processing apparatus configured to receive two or more correlation images, each representing an overlap between emitted and reflected light signals at a plurality of pixel locations of a corresponding input image at a given phase shift, the apparatus being configured to filter each correlation image in dependence on an expected relationship between the respective correlation signals of each correlation image at a given location in the input image, wherein the filtering is performed by means of a data driven model.
  • This may allow for performance of the filtering in an edge preserving manner and may allow for the detection of errors in the correlation images without using any ground truth.
  • the apparatus may be configured to perform the filtering in dependence on an expected physical relationship between the respective correlation signal of each correlation image at the given location.
  • the physical relationship between the correlation signals is preferably known.
  • the physical relationship between the correlation signals may be an assumed physical relationship.
  • the data driven model may be a neural network. This may be a convenient implementation.
  • the data driven model may be a trained artificial intelligence model.
  • the model may alternatively be a different type of model.
  • the model may or may not be based on deep learning.
  • the data driven model may be configured to be trained by means of the expected relationship in a self-supervised training procedure. This may improve the efficiency of the training of the model and may allow the model to be trained without the need for ground truth data, annotations or a second sensor.
  • the filtering of each correlation image may be so as to reduce noise in each correlation image. This may improve the quality of the correlation images and any subsequent data derived from them, such as a depth map.
  • the respective correlation signal may vary as a sine function or linear function of phase shift. Other functions may be possible.
  • the expected relationship may be determined for an area of the respective correlation image corresponding to multiple pixels of the input image.
  • Each area of the respective correlation image may correspond to an n x n pixel window of the input image. This may allow the most reasonable pixel to be used within the area of multiple pixels while preserving the original correlation relationship. This may allow for robustness to noise. For example, there might be an outlier pixel with an incorrect signal due to ambient light or surface properties, but looking at the surrounding pixels or window may still allow some meaningful information to be obtained for this pixel. It is also possible to see whether there is agreement between the pixels (local consistency for the statistics) and thus cure some misinformation.
  • the filtering may be performed by means of a learned kernel which is applied to the respective correlation signal of each correlation image at a location corresponding to a pixel of the input image.
  • the kernel may be learned using the data driven model, which may be a kernel prediction network. This may allow the output to be sharp whilst preserving the original information.
  • the kernel For each location corresponding to a pixel of the input image, the kernel may be applied to each correlation signal (i.e. if there are four correlation images, to each of the four correlation signals at the pixel location).
  • the apparatus may be configured to determine a depth map for the input image based on the filtered correlation images, the depth map representing an estimated distance of an object in the input image from an imaging device at a plurality of pixel locations in the input image. This may allow the filtered correlation images to be used in depth processing of the input image.
  • the apparatus may be configured to determine a confidence map for the input image based on the filtered correlation images, the confidence map representing an amount of uncertainty present in the estimated distances for each pixel in the input image.
  • the confidence map may advantageously be used in the depth fusion process. This may allow for training of a network with relatively poor-quality ground truth without harming details present in a depth map obtained from ToF data.
  • the apparatus may be further configured to, in dependence on the confidence map, generate an improved depth map for the input image.
  • the apparatus may be configured to fuse the depth map with the confidence map to generate the improved depth map. The physical relationship between the correlation images can thus be used to obtain an improved depth map for an input image.
  • the depth map may be determined from one or more greyscale or RGB images. This may allow the approach to be used in depth estimation for a variety of input images.
  • a method of filtering two or more correlation images of an input image in an image processing apparatus comprising filtering each correlation image in dependence on an expected relationship between the respective correlation signals of each correlation image at a given location in the input image, wherein the filtering is performed by means of a data driven model.
  • This may allow for performance of the filtering in an edge preserving manner and may allow for the detection of errors in the correlation images without using any ground truth.
  • the apparatus is configured to receive multiple correlation images. For example, there may be four correlation images.
  • the data driven model may be configured to be trained by means of the expected relationship in a self-supervised training procedure. This may improve the efficiency of the training of the model and may allow the model to be trained without the need for ground truth data, annotations or a second sensor.
  • a computer program which, when executed by a computer, causes the computer to perform the method described above.
  • the computer program may be provided on a non-transitory computer readable storage medium.
  • Figure 1 (a) schematically illustrates an example of multi-path interference.
  • Figure 1 (b) schematically illustrates an example of a motion artefact.
  • Figure 1 (c) schematically illustrates an example of sparsity and shot noise in correlation images.
  • Figure 2(a) shows an example of correlation images C1-C4.
  • Figure 2(b) schematically illustrates an example of a relationship between correlation images having a sinus assumption.
  • Figure 2(c) schematically illustrates an example of a relationship between correlation images having a linear assumption.
  • Figure 3 schematically illustrates an example of a KPN-based filtering architecture.
  • Figure 4 shows examples of results of correlation filtering using the method described herein compared to other approaches.
  • Figure 5 schematically illustrates an example of a depth fusion model architecture.
  • Figure 6 shows examples of results of depth fusion using the method described herein.
  • Figure 7 shows further examples of results of fusion using the method described herein compared to other approaches.
  • Figure 8 schematically illustrates an example of an image processing apparatus configured to implement the method described herein.
  • Embodiments of the present invention can filter correlation images using an expected physical relationship between the correlation images.
  • the correlation images are preferably produced from l-ToF data.
  • the filtering of each correlation image is so as to reduce noise in each correlation image, which may arise from one or more of multi path interference, motion artifact, shot noise etc.
  • the noise can be filtered out while preserving as many details as possible using the physical relationships between multiple correlation images at a given location in each of the correlation images.
  • the given location may correspond to the location of a pixel in each of the correlation images and may correspond to a pixel of a corresponding input image (for example, an RGB image captured by a camera). This allows for performance of the filtering in an edge preserving manner and allows for the detection of any error caused in the correlation images without using any ground truth.
  • this detected error can be used as a confidence map and combined with a dense, but not detailed, depth map predicted from the input image.
  • This two-step approach can allow for training of a completion network with relatively poor-quality ground truth without harming details present in a depth map obtained from l-ToF data.
  • the corrected correlation images can therefore be used to predict an analytic confidence map and can then be used for depth fusion with RGB image-based depth estimation, as will be described in more detail below.
  • the physical relationship between the correlation images can thus be used to obtain an improved depth map.
  • correlation images obtained from an I- ToF camera have a specific relationship. For example: Using this relationship, it is possible to detect errors in each pixel. Thus, the network can be trained to perform filtering by inspecting the relationship between the raw correlation signals.
  • Figure 2(a) shows a further example of four correlation images having correlation values or signals per pixel which vary as a function of phase shift.
  • the relationship between the correlation signals has a known relationship, such as sinus or linear. Other relationships may be possible.
  • the network is capable of performing edge preserving shot noise filtering and can return information regarding errors of each pixel caused by MPI or motion artefacts, which can be used for a second stage RGB depth fusion.
  • the network has the advantage of being very light-weight and does not require any ground truth, which is often hard to obtain, or may not be accurate enough.
  • KPN Kernel Prediction Network
  • the KPN predicts an (NxN) kernel per pixel which is then applied on the network's input to provide a final output. This allows the output to be sharp whilst preserving the original information.
  • the network 300 takes four stacked correlation images 301 as input and predicts a single 5x5 kernel per pixel, shown at 302, which is applied along the four correlation images with the same weight, as shown at 303.
  • the kernel can pick up the most reasonable pixel within a window of pixels while preserving the original correlation relationship.
  • the output filtered correlation images are shown at 304.
  • An analytical relationship error map 305 can optionally be determined.
  • This analytical error map 305 contains the amount of uncertainty present in the depth information from the correlation images, which can be subsequently used as a confidence map for depth sensor fusion, as will be described in more detail later.
  • the emitted modulated signal from 1-ToF is a sinus wave
  • the returned signal and corresponding correlation images will be a sinus function with parameter Y-
  • the sinus constraints for example or any other fixed relationship between sin(x) and cos(x)
  • the signal is a ra recentered such that it ranges from LU .
  • the real-life correlation signal can be modelled as shown in Eq.9.
  • h c w 'th ⁇ e 1, 2, 3, 4 symbolizes the noise which contains common baseline shift as well.
  • the common baseline factor By adding the four correlation images, the common baseline factor can be obtained, which can be used to change the range of the correlation images.
  • the first form of error can be measured by summing .
  • the second form of error can be obtained via the sinus relationship.
  • the relationship can be enforced to obtain the second form of error.
  • the expected physical relationship is determined for an area of the respective correlation image corresponding to multiple pixels.
  • each area of the respective correlation image corresponds to an n x n pixel window. This may allow the most reasonable pixel to be used within a window of pixels while preserving the original correlation relationship. This may allow for robustness to noise. For example, there might be an outlier pixel with an incorrect signal due to ambient light or surface properties, but looking at the window may still allow some meaningful information to be obtained for this pixel. It is also possible to see whether there is agreement between the pixels (local consistency for the statistics) and thus cure some misinformation.
  • the filtering is preferably performed by means of a learned kernel which is applied to the respective correlation signal of each correlation image at a location corresponding to a pixel.
  • the kernel is applied to each correlation signal (for example each of the correlation signals at that location in the correlation images C1 -C4).
  • the model was applied to real data by training on a real-world recording under ambient light with the given loss.
  • the initial output is sharp on regions where the raw signal is reliable.
  • the signal becomes more sparse where it is not reliable and predictions become noisy, but those regions can be detected reliably using the physical relationship such that the area can be thresholded out or suppressed during the fusion.
  • the depth map can therefore be fused with the help of both a learned RGB confidence map (504 in Figure 5, described below) and the analytic ToF error map (305 in Figure 3 and 508 in Figure 5, described below) to generate an improved depth map with improved resolution compared to the original RGB depth map.
  • the original depth map for the input image may be determined based on the filtered correlation images.
  • the depth map represents an estimated distance of an object in the form an imaging device at a plurality of locations of pixels.
  • a confidence map can be determined for the input image based on the filtered correlation images.
  • the confidence map represents an amount of uncertainty present in the estimated distances for each pixel in the input image.
  • the overall network's sparse branch is completely separated from the RGB branch and it is pre-trained with the physical relationship-based loss, which provides the analytic error from Eq.14.
  • FIG. 5 shows an overview of the network architecture.
  • a typical-encoder decoder architecture 501 may be used with skip connections (see Ronneberger et al. , ‘U-Net convolutional networks for biomedical image segmentation’, Medical Image Computing and Computer-Assisted Intervention, MICCAI 2015, p. 234-241 , Cham, 2015, Springer International Publishing).
  • the model is configured to receive an RGB input image 502 and output RGB depth 503 and the learned confidence 504 in the RGB depth.
  • the raw correlation images 505 are input to the data driven model 506.
  • the model is a light KPN.
  • the model outputs the filtered correlation images 507 and the analytic error map 508.
  • the RGB depth map 503 is fused and refined using the learned confidence 504, the filtered correlation images 507 and the analytic error map 508 to give a final depth map 510 for the RGB image.
  • a combination loss between ground truth depth d and predicted depth d and a depth gradient loss weighted by the image I's gradients for edge-awareness can be used (see Godard et al. , ‘Unsupervised monocular depth estimation with left-right consistency’, CVPR, 2017).
  • An exemplary loss function is shown below.
  • the depth map may alternatively be determined from more than one RGB image, or from one or more greyscale images.
  • FIG 6 shows an example of results obtained using this method.
  • the network was trained with an Intel RealSense depth camera (D435i) as ground truth.
  • D435i Intel RealSense depth camera
  • GT ground truth
  • the predictions were compared to the processed GT using the RMSE metric.
  • the prediction achieves 0.288m and 0.231 m with backward warping (i.e. warping the output) with and without occlusion error respectively, and 0.219m and 0.274m with forward warping (i.e. warping the GT) with and without occlusion error respectively.
  • RMSE 0.29m
  • the network was applied to an office scene and the quality of the depth prediction was visually analysed.
  • An example is shown in Figure 7.
  • the method described herein was shown to perform better, in particular in preserving sharp edges and thin details of objects whenever the raw signal is reliable (marked in a black rectangle). Thanks to the first stage of pre-training, the model recovers sharp details with the analytic confidence map which is preferably trained only with correlation relationships such that problematic GT values do not influence the output. For areas such as the floor or table top, where the raw signal is not reliable, the fusion network successfully completes the missing depth.
  • FIG 8 shows an example of a system 800 comprising an imaging device 801 configured to use the method describe herein to process image data captured by a plurality of image sensors 802, 803 in the device.
  • the device 801 has an RGB image sensor 802 and a sensor 803 configured to collect ToF depth data.
  • Such a device 801 typically includes some onboard processing capability. This could be provided by the processor 804.
  • the processor 804 could also be used for the essential functions of the device.
  • the transceiver 805 is capable of communicating over a network with other entities 810, 811 . Those entities may be physically remote from the device 801 .
  • the network may be a publicly accessible network such as the internet.
  • the entities 810, 811 may be based in the cloud.
  • Entity 810 is a computing entity.
  • Entity 811 is a command and control entity.
  • These entities are logical entities. In practice they may each be provided by one or more physical devices such as servers and data stores, and the functions of two or more of the entities may be provided by a single physical device.
  • Each physical device implementing an entity comprises a processor and a memory.
  • the devices may also comprise a transceiver for transmitting and receiving data to and from the transceiver 805 of device 801 .
  • the memory stores in a nontransient way code that is executable by the processor to implement the respective entity in the manner described herein.
  • the command and control entity 811 may train the models used in the apparatus. This is typically a computationally intensive task, even though the resulting model may be efficiently described, so it may be efficient for the development of the algorithm to be performed in the cloud, where it can be anticipated that significant energy and computing resource is available. It can be anticipated that this is more efficient than forming such a model at a typical imaging device.
  • the command and control entity can automatically form a corresponding model and cause it to be transmitted to the relevant imaging device.
  • the method is implemented at the device 801 by processor 804.
  • an image may be captured by the sensors 802, 803 and the image data may be sent by the transceiver 805 to the cloud for processing. The resulting image could then be sent back to the device 801 , as shown at 812 in Figure 8.
  • the method may be deployed in multiple ways, for example in the cloud, on the device, or alternatively in dedicated hardware.
  • the cloud facility could perform training to develop new models or refine existing ones.
  • the training could either be undertaken close to the source data, or could be undertaken in the cloud, e.g. using an inference engine.
  • the method may also be implemented at the device, in a dedicated piece of hardware, or in the cloud.
  • Embodiments of the present invention may therefore advantageously use an RGB image and correlation images as input and exploit the physical relationship of the correlation images to improve the depth fusion to give an improved depth map.
  • the apparatus can be configured to fuse the depth map with the confidence map to generate the improved depth map.
  • the depth map and corrected correlation images may be fused with a combined confidence map with information from both input modalities.
  • the model used to perform the correlation correction is preferably a neural network.
  • the correlation correction may also be performed by means of another trained artificial intelligence model or a data driven model without deep learning.
  • the model can advantageously be trained by means of the expected relationship between the correlation images in a self- supervised fashion, without the need for ground truth data, annotations or a second sensor.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Electromagnetism (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un appareil de traitement d'image (800) configuré pour recevoir au moins deux images de corrélation (301, 505), chacune représentant un chevauchement entre des signaux lumineux émis et réfléchis au niveau d'une pluralité d'emplacements de pixels d'une image d'entrée correspondante (502) à un déphasage donné, l'appareil (800) étant configuré pour filtrer chaque image de corrélation en fonction d'une relation attendue entre les signaux de corrélation respectifs de chaque image de corrélation à un emplacement donné dans l'image d'entrée (502), le filtrage étant effectué au moyen d'un modèle entraîné par des données (506). Ceci peut permettre la réalisation du filtrage d'images de corrélation dans un mode de préservation de bord et peut permettre la détection d'erreurs dans les images de corrélation sans utiliser de vérité de terrain.
PCT/EP2021/056657 2021-03-16 2021-03-16 Appareil et procédé de correction de corrélation d'images Ceased WO2022194352A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/EP2021/056657 WO2022194352A1 (fr) 2021-03-16 2021-03-16 Appareil et procédé de correction de corrélation d'images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2021/056657 WO2022194352A1 (fr) 2021-03-16 2021-03-16 Appareil et procédé de correction de corrélation d'images

Publications (1)

Publication Number Publication Date
WO2022194352A1 true WO2022194352A1 (fr) 2022-09-22

Family

ID=75108326

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2021/056657 Ceased WO2022194352A1 (fr) 2021-03-16 2021-03-16 Appareil et procédé de correction de corrélation d'images

Country Status (1)

Country Link
WO (1) WO2022194352A1 (fr)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8265425B2 (en) 2008-05-20 2012-09-11 Honda Motor Co., Ltd. Rectangular table detection using hybrid RGB and depth camera sensors
US8786678B2 (en) 2011-01-12 2014-07-22 Sony Corporation 3D time-of-flight camera and method
US20160005179A1 (en) 2012-12-17 2016-01-07 Lsi Corporation Methods and apparatus for merging depth images generated using distinct depth imaging techniques
US20160232684A1 (en) * 2013-10-18 2016-08-11 Alexander Borisovich Kholodenko Motion compensation method and apparatus for depth images
US9424650B2 (en) 2013-06-12 2016-08-23 Disney Enterprises, Inc. Sensor fusion for depth estimation
US9538162B2 (en) 2012-02-29 2017-01-03 Samsung Electronics Co., Ltd. Synthesis system of time-of-flight camera and stereo camera for reliable wide range depth acquisition and method therefor
US20180067197A1 (en) * 2015-04-01 2018-03-08 Iee International Electronics & Engineering S.A. Method and system for real-time motion artifact handling and noise removal for tof sensor images
US10311378B2 (en) 2016-03-13 2019-06-04 Microsoft Technology Licensing, Llc Depth from time-of-flight using machine learning
US20210004937A1 (en) * 2019-07-02 2021-01-07 Microsoft Technology Licensing, Llc Machine-learned depth dealiasing

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8265425B2 (en) 2008-05-20 2012-09-11 Honda Motor Co., Ltd. Rectangular table detection using hybrid RGB and depth camera sensors
US8786678B2 (en) 2011-01-12 2014-07-22 Sony Corporation 3D time-of-flight camera and method
US9538162B2 (en) 2012-02-29 2017-01-03 Samsung Electronics Co., Ltd. Synthesis system of time-of-flight camera and stereo camera for reliable wide range depth acquisition and method therefor
US20160005179A1 (en) 2012-12-17 2016-01-07 Lsi Corporation Methods and apparatus for merging depth images generated using distinct depth imaging techniques
US9424650B2 (en) 2013-06-12 2016-08-23 Disney Enterprises, Inc. Sensor fusion for depth estimation
US20160232684A1 (en) * 2013-10-18 2016-08-11 Alexander Borisovich Kholodenko Motion compensation method and apparatus for depth images
US20180067197A1 (en) * 2015-04-01 2018-03-08 Iee International Electronics & Engineering S.A. Method and system for real-time motion artifact handling and noise removal for tof sensor images
US10311378B2 (en) 2016-03-13 2019-06-04 Microsoft Technology Licensing, Llc Depth from time-of-flight using machine learning
US20210004937A1 (en) * 2019-07-02 2021-01-07 Microsoft Technology Licensing, Llc Machine-learned depth dealiasing

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
GODARD ET AL.: "Unsupervised monocular depth estimation with left-right consistency", CVPR, 2017
GUO ET AL.: "Tackling 3d ToF artifacts through learning and the flat dataset", THE EUROPEAN CONFERENCE ON COMPUTER VISION (ECCV, September 2018 (2018-09-01)
LEE ET AL.: "Sensors, Cameras and Systems for Industrial and Scientific Applications XIII", vol. 8298, 2012, INTERNATIONAL SOCIETY FOR OPTICS AND PHOTONICS, article "Motion blur-free time-of-flight range sensor", pages: 82980U
MILDENHALL ET AL.: "Burst denoising with kernel prediction networks", 2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2018, pages 2502 - 2510, XP033476216, DOI: 10.1109/CVPR.2018.00265
RONNEBERGER ET AL.: "Medical Image Computing and Computer-Assisted Intervention, MICCAI 2015", 2015, SPRINGER INTERNATIONAL PUBLISHING, article "U-Net convolutional networks for biomedical image segmentation", pages: 234 - 241
SU ET AL.: "Deep end-to-end time-of-flight imaging", PROCEEDINGS OF THE IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2018, pages 6383 - 6392, XP033473555, DOI: 10.1109/CVPR.2018.00668
WEN YANG ET AL: "Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution", IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE, USA, vol. 28, no. 2, 1 February 2019 (2019-02-01), pages 994 - 1006, XP011701957, ISSN: 1057-7149, [retrieved on 20181023], DOI: 10.1109/TIP.2018.2874285 *
ZHANG ET AL.: "The IEEE Conference on Computer Vision and Pattern Recognition (CVPR", 2018, article "Deep Depth Completion of a Single rgb-d Image"

Similar Documents

Publication Publication Date Title
US12340556B2 (en) System and method for correspondence map determination
Liu et al. Planemvs: 3d plane reconstruction from multi-view stereo
Shin et al. Estimation of ambient light and transmission map with common convolutional architecture
Skinner et al. Uwstereonet: Unsupervised learning for depth estimation and color correction of underwater stereo imagery
US11215700B2 (en) Method and system for real-time motion artifact handling and noise removal for ToF sensor images
CN114746895A (zh) 用于图像去噪的噪声重构
CN112529015A (zh) 一种基于几何解缠的三维点云处理方法、装置及设备
CN109640066B (zh) 高精度稠密深度图像的生成方法和装置
CN110400338B (zh) 深度图处理方法、装置和电子设备
KR102809044B1 (ko) 영상의 깊이를 추정하는 방법 및 장치
CN115578704A (zh) 深度估计模型训练、深度估计方法、装置、设备及介质
US20230393278A1 (en) Electronic device, method and computer program
O'Byrne et al. A stereo‐matching technique for recovering 3D information from underwater inspection imagery
Li et al. Real‐Time Visualization System for Deep‐Sea Surveying
US12125223B2 (en) Time-of-flight depth enhancement
WO2022194352A1 (fr) Appareil et procédé de correction de corrélation d'images
Wu et al. Disparity estimation using a quad-pixel sensor
Zuo et al. MonoTher-Depth: Enhancing Thermal Depth Estimation via Confidence-Aware Distillation
Choi et al. Implementation of Real‐Time Post‐Processing for High‐Quality Stereo Vision
Yang et al. End-to-end underwater multi-view stereo for dense scene reconstruction
Liu et al. Three dimensional moving pictures with a single imager and microfluidic lens
Ochotorena et al. Gradient-guided filtering of depth maps using deep neural networks
Du et al. Towards High-Precision Depth Sensing via Monocular-Aided iToF and RGB Integration
CN118644562B (zh) 一种模型训练方法、三维点云获取方法及电子设备
CN119206124B (zh) 一种基于稀疏视图的新视角合成方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21712992

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21712992

Country of ref document: EP

Kind code of ref document: A1