WO2024194540A1 - Image correction method - Google Patents

Abstract

Disclosed is a method for rendering at least one source image of a scene obtained by an optical sensor of an imaging system. The method comprises a phase of processing a reconstructed image, at least some of the pixels of which includes depth information and/or speed information. At the end of the processing phase, a rendered image R is obtained for each processed reconstructed image. The processing phase includes an iterative modification of the reconstructed image so as to minimize a function E comprising a term D, called difference term, determined by comparing the reconstructed image convolved by a point spread function (PSF) with the source image and a term A, called penalty term, which is representative of defects or anomalies in the reconstructed image and is determined from the reconstructed image.

Description

DESCRIPTION

TITLE: Image correction process

Technical field

The present invention relates to the field of image processing and correction methods. These methods aim to improve the overall quality or overall rendering of an image containing defects.

The invention relates to a method for correcting and improving the quality of any type of image.

The invention relates in particular, but not exclusively, to any apparatus comprising cameras or optical sensors capable of acquiring images. For example, it may be photographic devices, cameras, smart phones (or smartphones), tablets, laptops or systems, vehicles or infrastructures equipped with cameras.

State of the prior art

The best known and most widespread image processing method is so-called stereo vision, derived from binocular human vision, or more broadly animal vision, from two images that are compared. The biological processing of the images obtained is extremely efficient, since it provides in real time a notion of depth of the observed scene, allowing for example to move knowing at what relative distance the observed objects are.

Despite these remarkable performances, this biological stereo vision system does not allow to simultaneously restore a sharp image over a large depth of field, ranging from a few tens of centimeters to infinity. In particular, this blur comes partly from the limited depth of field of the human eye. The ocular optics do not allow to obtain a focus simultaneously at all distances, which is well explained by the conjugation law in geometric optics. This well-known phenomenon is linked to the notion of accommodation of the eye. But also, human vision involves a rotational displacement of the eyes in particular to match the zone of best sharpness of each eye (fovea), which depends on the distance of the observed objects. As a result, objects observed outside the distance of best focus may not overlap correctly and give rise to an imperfect 3D interpretation. Also known in the state of the art are methods for reconstructing images comprising relief or depth information from 2D images. These methods make it possible to deduce depth information associated with the images from two images. A widely used reconstruction technique is photogrammetry. However, there are several problems and limitations to the reconstruction methods of the state of the art. Indeed, all these techniques require identifying the most common pixels on the 2D images from which the depth is determined. This identification is necessary to be able to extract the depth information.

Therefore, to obtain information on a large part of an imaged scene and/or to obtain the most reliable depth information possible, state-of-the-art methods require images with the highest possible definition as well as the highest possible resolution.

However, despite good image quality and/or definition, when the 2D images from which the depth is determined include smooth areas, i.e. areas with little relief, or areas with little or too much texture or areas with little or too much contrast, the depth information determined is at best erroneous or even impossible to determine.

Furthermore, the raw images obtained by optical systems equipping image acquisition devices, and in particular consumer devices such as cameras and smartphones, are imperfect and contain defects inherent to image acquisition devices. Although there are image processing methods aimed at improving their quality, defects persist in the images processed using state-of-the-art methods and/or the resolution of the corrected image is reduced compared to the raw image obtained by the sensor. These two effects add to the problems mentioned above and induce additional errors in the extracted depth information and prevent the depth information from being determined on a large number of pixels of the reconstructed images. Finally, state-of-the-art image processing methods, when implemented by processing means having “standard” computing power, often have a high processing time which is not compatible with the extraction of depth information in real time, in particular from a continuous image stream. Most optical systems equipping cameras and smartphones have a single focusing system and a single photographic sensor. These optical systems generate mosaicked raw images due to the use of a matrix of colored filters, the best known example being the Bayer matrix, allowing the separation of colors. This technique requires that the different color channels, typically R, G, B (Red, Green, Blue), are obtained at different positions of the photographic sensor, i.e. via different photosites, typically photodiodes, forming a mosaic. As a result, the colors rendered on the raw image coming from the sensor are not rendered at all points (photosites) of the sensor. In order to maintain the best possible resolution, i.e. the native resolution of the sensor corresponding to the tiling of the photosites, it is necessary to process the raw image to extract the two missing colors from the photosites, for example blue and green for the red photosite. Methods are known in the state of the art for removing this mosaic effect from raw images. However, these methods result in the appearance of color fringes and the erroneous restitution of colors not present in the imaged object.

Other cameras and smartphones are equipped with optical systems comprising several separate focusing lenses and a stack of several aligned photosite planes in the sensor. Thus, each pixel of the raw image comprises the three color channels and no mosaic effect has to be processed. However, these optical systems generate aberrations on the raw image linked to focusing effects and defects. In practice, this generates geometric deformations and blurring effects. Methods are known in the state of the art to eliminate these optical and chromaticity aberrations. However, these methods lead to the appearance of restitution errors and defects within the processed image.

One aim of the invention is to propose an image processing method enabling:

- to improve the quality of images, and/or

- to compensate for focusing defects inherent in any optical lens, and/or

- to limit artifacts related to the projection of the image onto the color detection matrix, and/or

- to limit the effects of noise, and/or

- to increase the dynamic range in amplitude, and/or - to extract a depth, i.e. the distance to the objective, and/or a speed at each point of the image from images not including depth or speed information, and/or

- to improve the depth and/or speed information contained in an image, and/or

- to increase the depth of field of the image, without resorting to a focus adjustment of the optical image acquisition system,

- to process and/or enrich images and/or extract depth and/or speed information more precisely than the state of the art.

Presentation of the invention

For this purpose, a method is proposed for restoring at least one image from one or more source images. Each source image is an image of a scene obtained by an optical sensor of an imaging system.

The method comprises a phase of processing a reconstructed image, associated with a source image, at least some of the pixels of which comprise depth information and/or speed information. At the end of the processing phase, a restored image is obtained for each reconstructed image processed. The processing phase comprises an iterative modification of the reconstructed image so as to minimize a function E.

The function E includes a term D, called the difference term, determined by comparing the reconstructed image convolved by a point spread function (PSF) with the source image, to which the reconstructed image is associated. The PSF describes the response of the imaging system from which the source image used for comparison is obtained.

The function E includes a term A, called anomalies or penalties, representative of defects or anomalies within the reconstructed image, determined from the reconstructed image.

Preferably, the method comprises, prior to and/or, respectively, during the processing phase, a step of reconstructing the reconstructed image or respectively the reconstructed image currently being processed from at least one source image and/or at least one reconstructed image and/or at least one reconstructed image currently being processed and/or at least one restored image. Preferably, the step of reconstructing the reconstructed image, or respectively the image being reconstructed, comprises or is carried out by or consists of an addition of depth and/or speed information and/or a substitution of the depth information and/or the speed information contained in at least a portion of the pixels of the at least one source image and/or of the at least one reconstructed image and/or of the at least one reconstructed image being processed and/or of the at least one restored image, or respectively in at least a portion of the pixels of the at least one reconstructed image being processed.

Preferably:

- the depth information and/or the speed information added or substituted is obtained from or by comparing the reconstructed image or the reconstructed image being processed with at least one source image and/or at least one reconstructed image and/or at least one reconstructed image being processed and/or at least one restored image, and/or

- the depth information and/or the speed information added or substituted is obtained by comparing at least one source image and/or at least one reconstructed image and/or at least one reconstructed image being processed and/or at least one restored image with at least one source image and/or at least one reconstructed image and/or at least one reconstructed image being processed and/or at least one restored image, and/or

- depth information and/or speed information is added or substituted by moving a pixel or a set of pixels or one or more blocks of an image by an operator, for example differential or vector, and/or according to a function, for example a scalar or vector field.

In this application, the term "image being processed" may be understood to mean the reconstructed image being processed or being restored. In this application, the term "processing phase" may be understood to mean the iterative modification of the reconstructed image.

In the present application, the term "image" used alone may mean a source image, a reconstructed image, a reconstructed image during processing or a rendered image. In the present application, "image block" may be understood to mean a set of pixels, preferably contiguous and/or adjacent to each other, of an image. In the present application, imaging system may be understood to mean an optical system capable of acquiring images. For example, the imaging system may be a camera.

The processing phase can be defined as consisting of minimizing the function E by modifying the reconstructed image. In other words, the processing phase can be defined as comprising at least one iteration of a step of modifying the reconstructed image aimed at minimizing the function E.

The reconstruction step may be defined as consisting of adding depth information and/or speed information, for example when the latter is missing, in at least a portion of the pixels of an image not containing depth information and/or speed information and/or substituting depth information and/or speed information contained in a source image, a reconstructed image, a reconstructed image being processed or a restored image and/or moving a pixel or a set of pixels or one or more blocks of an image by one and/or according to a function.

The image obtained by such a displacement of a pixel or a set of pixels or of one or more blocks of an image can be considered as a reconstructed image in that the image before displacement sees, among other things, the depth information and/or the speed information contained in one or more pixels modified in the image obtained after displacement. As an example, the reconstruction step can comprise a displacement of one or more pixels or of a set of pixels or of one or more blocks of an image according to a speed field. Preferably, the speed field corresponds to or is representative of a relative movement of one, several or each of the objects of the scene with respect to the imaging system as a function of time.

The PSF can be substituted by any known function describing the response of an imaging system. For example, the PSF can be substituted by an optical transfer function (OTF) of the imaging system.

Preferably, the function E, more preferably the term D and/or the term A, depends on the speed information and/or the depth information.

Preferably, the method or step of reconstructing the reconstructed image comprises extracting the depth information and/or the speed information from the imaged scene by comparing at least two source images or at least two reconstructed images or at least two reconstructed images in progress. processing or at least two rendered images or at least one source image with at least one rendered image or with one reconstructed image or with one reconstructed image being processed.

In the present application, depth extraction and/or speed information by image comparison may consist of or include a comparison of at least two source images or at least two reconstructed images or at least two reconstructed images being processed or at least two rendered images or a comparison of at least any two images among at least one source image, at least one rendered image, at least one reconstructed image and at least one reconstructed image being processed.

The reconstruction method or step may include:

- during the processing phase, an addition of depth information and/or speed information in at least a portion of the pixels of the reconstructed image being processed not containing depth information and/or speed information and/or a substitution of the depth information and/or speed information contained in the at least a portion of the pixels of the reconstructed image being processed from the depth information and/or speed information extracted from an image comparison, and/or

- prior to the processing phase, an addition of depth information and/or speed information in at least part of the pixels of an image not containing depth information and/or speed information and/or a substitution of the depth information and/or speed information contained in the at least part of the pixels of the image from the depth information and/or speed information extracted from an image comparison.

Preferably, the term D is determined by comparison of:

- the reconstructed image or the reconstructed image being processed convolved with the PSF, with

- the source image, to which the reconstructed image is associated.

Preferably, the step of determining the term D comprises, prior to the comparison of the convolved reconstructed image with the PSF or the reconstructed image being processed convolved with the PSF with the source image, with which the reconstructed image is associated, a spreading, at least locally, according to a speed field and/or according to an acquisition duration of said source image, with which the reconstructed image or the reconstructed image being processed is associated, of pixel(s) or group(s) or block(s) of pixels of said reconstructed image convolved with the PSF or of said reconstructed image being processed convolved with the PSF.

Preferably, the spreading, preferably local, of pixel(s) or group(s) or block(s) of pixels of the reconstructed image convolved with the PSF or of the reconstructed image being processed convolved with the PSF is carried out according to a velocity field.

Preferably, the processing phase comprises, prior to the convolution of the reconstructed image or the reconstructed image being processed with the PSF, a selection of the PSF.

Preferably, the reconstruction step comprises a convolution of the source image, to which the reconstructed image is associated, by an inverse function of the PSF, called iPSF, describing the response of the imaging system from which the source image, convolved by the iPSF, is obtained.

Preferably, the iterative modification of the reconstructed image ends when the function E, or a combination of partial derivatives of the function E with respect to the reconstructed image being processed, is less than a minimization threshold, or when a certain number of iterations of the iterative modification of the reconstructed image is reached; the at least one reconstructed image thus modified is restored.

Preferably, the reconstruction step comprises or is implemented by or is performed by or consists of a displacement of blocks, according to a speed field, of a source image, of a reconstructed image, of a reconstructed image during processing or of a restored image.

Preferably, the reconstruction step comprises or is implemented by or is performed by or consists of a displacement of blocks, according to a depth field, of a reconstructed image, of a reconstructed image being processed or of a restored image. Preferably, the reconstructed image is obtained by merging at least two reconstructed images.

Preferably, the processing phase comprises a step of merging at least two rendered images to provide at least one rendered image.

Preferably, for a considered imaging system, the PSF depends on:

- an orientation of an optical axis of the imaging system considered relative to at least one other optical axis of another imaging system, and/or

- a relative position of the imaging system considered with respect to at least one other imaging system, and/or

- a distance between the sensor of the imaging system considered and an object imaged by the sensor of the imaging system considered, and/or

- a distance between a part of the imaging system considered and the at least one sensor, and/or

- a distance between a part of the imaging system considered and an object imaged by the sensor of the imaging system considered, and/or

- a state of the imaging system, such as a zoom or focus or numerical aperture setting of the imaging system, and/or

- of the pixel or of a group or of a block of pixels of a source image or of a reconstructed image or of a restored image and/or of a photosite of the sensor of the imaging system considered, and/or

- of one or more pieces of information contained in a pixel or a group or block of pixels of an image, and/or

- one or more angles between the optical sensor of the imaging system considered and the imaging system.

Preferably, the PSF is selected from a set of PSFs. Preferably, the PST is selected based on one or more of the parameters on which the PSF depends.

Preferably, the at least one source image I comprises a set of source images comprising or consisting of a temporal succession of source images (Itl, It2...) acquired, preferably by the same imaging system, at different times (tl, t2...), each processing phase (Pl, P2...), each reconstruction step (Ml, M2...), each reconstructed image (Ml, M2...) and each restored image (Cl, C2...) are associated with a source image obtained at a given time t.

Preferably, the reconstructed image Mt is obtained from the source image It with which the reconstructed image Mt is associated.

The reconstructed image Mt may not be reconstructed from the source image It with which it is associated. For example, the reconstructed image Mt may be obtained or reconstructed from at least one reconstructed image and/or at least one reconstructed image currently being processed and/or at least one restored image, associated with a source image (It-1, It-2...) obtained at a previous time (t-1, t-2...).

Preferably, the reconstructed image Mt, associated with the source image It, is obtained, or the step of reconstructing the image Mt, associated with the source image It, comprises or is carried out, by comparison:

- of an image obtained by displacements, preferably according to a speed field and/or according to a depth field, of blocks of the restored image Ct-1, or of the reconstructed image Mt-1 or of the reconstructed image Mt-1 currently being processed, at the processing phase Pt-1, with

- the source image It, to which the reconstructed image Mt is associated, convolved by the inverse function of the PSF, called iPSF, describing the response of the imaging system from which the source image It is obtained.

Preferably, the reconstructed image mt, associated with the source image It, is obtained from the restored image Ct-1 obtained in the processing phase Pt-1.

Preferably, the reconstructed image Mt at the processing phase Pt is obtained by moving blocks, according to a velocity field, of:

- the restored image Ct-1 obtained in the processing phase Pt-1 or of the reconstructed image Mt-1 in the processing phase Pt-1 or of the reconstructed image Mt-1 currently being processed in the processing phase Pt-1, with which the source image It-1 is associated, or

- from the source image It, to which the reconstructed image Mt is associated.

Preferably, at least one step, preferably each step, of the method, and in particular the processing phase and the reconstruction step, is implemented for each source image (Itl, It2...) acquired at each of the different times (tl, t2...). Preferably, the at least one source image I comprises a set of source images acquired by different imaging systems (d, g...), preferably at the same given time or instant t, each processing phase (Pd, Pg...), each reconstructed image (Md, Mg...) and each restored image (Cd, Cg...) are associated with a source image I, of the set of source images (Id, Ig...), acquired by a different imaging system; the imaging systems being arranged so as to acquire, at least in part, preferably mainly, the same scene.

Preferably, at least one step, preferably each step, of the method, and in particular the processing phase and the reconstruction step, is implemented for each source image (Id, Ig...) acquired by a different imaging system (d, g...).

Preferably, the step of reconstructing the reconstructed image M is carried out from:

- at least one source image Id or at least one reconstructed image Md or at least one restored image Cd, and

- at least one source image Ig or at least one reconstructed image Mg or at least one restored image Cg.

Preferably, the temporal succession of source images (Itl, It2...) and the source images acquired by the different imaging systems (d, g...) can form two distinct sets of source images.

Preferably, the temporal succession of source images (Itl, It2, etc.) and the source images acquired by the different imaging systems (d, g, etc.) form the same set of source images (Idtl, Idt2, Igtl, Igt2, etc.). Preferably, each of the source images of the set of source images (Idtl, Idt2, Igtl, Igt2, etc.) is acquired by a separate imaging system (d, g, etc.) at a given time (tl, t2, etc.). Each processing phase (Pdtl, Pdt2, Pgtl, Pgt2, etc.), each reconstructed image (Mdtl, Mdt2, Mgtl, Mgt2, etc.) and each restored image (Cdtl, Cdt2, Cgtl, Cgt2, etc.) are associated with a source image obtained by a given imaging system and at a given time t. According to the invention, there is also provided a device or an apparatus or a data processing system comprising means, for example a processing unit, arranged and/or programmed and/or configured to implement the method according to the invention.

According to the invention, there is also provided a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to the invention.

According to the invention, there is also provided a medium, for example a recording medium, readable by a computer comprising instructions which, when executed by a computer, lead the latter to implement the method according to the invention.

According to the invention, a computer-readable data carrier is also provided, on which the computer program according to the invention is recorded.

Description of figures

Other advantages and particularities of the invention will appear on reading the detailed description of implementations and embodiments which are in no way limiting, and the following appended drawings: [Fig. 1] FIGURE 1 is a diagram illustrating the implementation of the iterative loop, consisting of successive modifications or successive iterations of modifications made to the reconstructed image, during the processing phase, for each of the source images acquired successively, by an imaging system, [Fig. 2] FIGURE 2 is a schematic representation of the iterative loop of the processing phase implemented from the source images of the left imaging system,

[Fig. 3] FIGURE 3 shows a simplified overall view of the image restitution method according to the invention,

[Fig. 4] FIGURE 4 illustrates an embodiment of the initialization step of the method, [Fig. 5] FIGURE 5 illustrates an embodiment of the initialization step of the method,

[Fig. 6] FIGURE 6 illustrates one embodiment of depth extraction from two images.

Description of the embodiments

The embodiments described below being in no way limiting, it will be possible in particular to consider variants of the invention comprising only a selection of features described, isolated from the other features described (even if this selection is isolated within a sentence comprising these other features), if this selection of features is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection comprises at least one feature, preferably functional without structural details, or with only a part of the structural details if this part only is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

With reference to FIGURES 1 to 6, an embodiment of the method 1 for restoring images from source images I according to the invention is presented. Each source image I is an image of a scene obtained by an optical sensor of an imaging system 2, for example a camera 2. Preferably, the method 1 is implemented for a set of source images I. The method can be implemented from source images I stored on a storage medium or can be implemented, in real time, from a stream of source images I coming from one or more image acquisition devices 2, such as cameras 2.

With reference to FIGURE 1, the method 1 is implemented for a set of source images consisting of a temporal succession of source images ItO, Itl, ..., It-1, It, It+1... acquired, by the same imaging system 2, at different times tO, tl ... t-1, t, t+1.... In FIGURE 1, the times at which each source image I is acquired are discretized tO, tl, ..., t-1, t, t+1... so that each new source image ItO, Itl, ..., It-1, It, It+1 acquired is incremented successively. However, the method 1 can also be implemented for a set of source images I acquired by different imaging systems 2 d, g... at the same time t. According to the invention, the imaging systems 2 are arranged so as to acquire the same scene.

The method 1 according to the invention is particularly suitable for processing a set of source images IdtO, Idtl, ..., Idt-1, Idt, Idt+1..., IgtO, Igtl, ..., Igt-1, Igt, Igt+1... acquired by a separate imaging system 2 d, g... at a given time tO, tl, t-1, t, t+1.... Thus, the method 1 makes it possible to process in real time a stream of source images Idt, Igt each originating from several imaging systems 2 (d, g). This embodiment is particularly suitable, among other things, for processing images which is implemented by autonomous vehicles. However, it is also particularly suitable for any type of systems or devices, such as, among others, smartphones and tablets, which have several separate cameras. In the case of autonomous vehicles, several cameras 2 are arranged in different positions to acquire images of the surroundings of the vehicle. In general, several cameras are oriented and arranged so as to acquire the same scene. For the sake of simplicity and understanding, the description of the embodiment is based on the use of two imaging systems 2 called right, noted d, and left, noted g. The two imaging systems 2 right and left are arranged so as to acquire, at least in part, the same scene.

Also, with reference to FIGURE 3, and in accordance with the non-limiting embodiment presented, the image restitution method 1 comprises a processing phase Pt of a reconstructed image Mdt, Mgt. As illustrated in FIGURE 2 and detailed below, the processing phase Pt is implemented independently for each reconstructed image Mdt, Mgt and each reconstructed image Mdt, Mgt is associated with a distinct source image Idt, Igt; each source image Idt, Igt being obtained by a distinct imaging system 2 d, g. At the end of the processing phase Pt, a restored image Cdt, Cgt is obtained for each reconstructed image Mdt, Mgt processed. As illustrated in FIGURES 1 and 2, the processing phase Pt comprises an iterative modification of the reconstructed image Mdt, Mgt so as to minimize a function E.

The processing phase can therefore be described as consisting of iterating the processing loop 3 or iterative loop 3 illustrated in FIGURE 2. With reference to FIGURE 1, the number of iterations is denoted k. For the processing loop Pt-1, the reconstructed image Mt-1 is modified iteratively until the function E, or a combination of partial derivatives of the function E with respect to the reconstructed image Mt-1 being processed, is less than a minimization threshold. In other words, the reconstructed image M will be modified at each iteration k of the iterative loop 3 so that the reconstructed image M being processed (obtained after each iterative modification), and ultimately the restored image C, best matches or is the most consistent or best restores the observed scene. The reconstructed image M before being modified for the first time in the first iterative loop 3 (or during the first iterative modification of the processing phase P) can be called reconstructed image M or reconstructed image M at iteration k=0, the reconstructed image M before being modified for the second time (i.e. after the first iterative loop 3 of the processing phase P and before and before the second iterative modification of the processing phase P) can be called the reconstructed image M being processed at iteration k>0 (in the case described at iteration k=l). In other words, the reconstructed image M (at iteration k=0) corresponds to the reconstructed image that has never been modified in the iterative loop 3 and the reconstructed image M being processed (at iteration k>0) corresponds to the set of reconstructed images M being processed at iterations k>0 that are being restored or are being iteratively modified. It can also be provided that the processing phase P is iterated until a certain number of iterations k is reached in the event that the minimization threshold cannot be reached in a satisfactory time, and therefore number of iterations. At the processing phase Pt-1, the reconstructed image Mt-1 thus modified after kl iterations is restored Ct-1. At the processing phase Pt, the reconstructed image Mt as modified after k2 iterations is restored (restored image Ct). Also, method 1 comprises a succession of processing phases P, each implemented for a different source image I. In other words, method 1 comprises the implementation of as many processing phases P as there are processed source images I.

The function E includes a term D, called the difference or anomaly term, determined by comparing the reconstructed image Mdt, Mgt convolved by a point spread function (PSF) with the source image Idt, Igt, to which the reconstructed image Mdt, Mgt is associated. Thus, each processing phase Pt (and therefore each reconstructed image Mt, each reconstructed image Mt being processed and each restored image Ct is associated with the source image It to which the reconstructed image Mt convolved by the PSF is compared during the processing phase Pt. The PSF describes the response of the imaging system d or g from which the source image Idt, Igt used for the comparison is obtained. The comparison step consists in detecting any information or value or data contained in the pixels of the pixel matrix of the compared images. According to an advantageous but non-limiting embodiment, the comparison step consists in detecting the colors within a color matrix, for example a Bayer matrix, of the reconstructed image Mt and the source image It and in comparing them in order to find the best match. In order to provide improved and/or precise depth information, it is advantageous to take into account only the pixels R (Red), VI (Green, or G (Green)), V2 (Green, or G (Green)), B (Blue) of the Bayer matrix corresponding to the reconstructed image Mt, or on pixels without a color filter with a particular geometry, called "dual pixels", to provide sensitivity to the depth of the image for the auto-focusing of the objective of the imaging system 2. However, the comparison step may consist of detecting a or several data contained in the pixels of the pixel matrix of the images compared among a color data and/or texture data, and/or brightness data, and/or hue data, and/or saturation data, and/or RGB data.

The function E includes a term A, called a penalty term, representative of defects or anomalies within the reconstructed image Mdt, Mgt, determined from the reconstructed image Mdt, Mgt.

The P processing phase is described in detail below.

Each image (source I, reconstructed M, reconstructed M in the process of processing and restored C) can be represented in a table of values. It contains at discretized positions color values, for example according to the three colors R (red), G (green), B (blue), or in another reference frame. We can see these elements as juxtaposed square or rectangular elementary objects, comprising three color properties, in addition to width and length properties of the elements common to all these elementary objects, as well as the position property on the optical sensor implicitly known by a row arrangement in the table and at a constant pitch in each direction x and y (x and y being two orthogonal directions in the image plane). These objects can be called pixels. Also, with the exception of the source images I which do not contain depth information, each reconstructed image Mdt, Mgt, each reconstructed image Mdt, Mgt being processed and each rendered image Cdt, Cgt is representable by a pixel table or pixel matrix containing (xd or xg, yd or yg, Z). The reconstructed images Mdt, Mgt, reconstructed images Mdt, Mgt being processed and rendered images Cdt, Cgt containing depth information can be described as images enriched with depth information. Furthermore, the objects in the scene imaged by the imaging systems 2 are at different distances from one or more optical sensors (of the imaging systems 2) and therefore at a different depth. Also, a group or block of pixels corresponding to one of the objects of the imaged scene will have depth information different from one or more other groups or blocks of pixels corresponding to the other or other objects of the imaged scene. Consequently, each image will be enriched by a depth field, that is to say that each pixel or group or block of pixels (of the matrix or table of pixels representing the image of the scene) corresponding to an object of the scene will have depth information which corresponds to the distance between the object and the optical sensor from which the source image I was acquired; the depth of the different objects of the imaged scene being, in the most cases, different for at least some of the objects in the scene. The person skilled in the art will know how to convert or transpose the coordinates or positions of the pixels (or any other associated operator, matrix or grid) from the x and y space (sensor reference frame) to another spatial reference frame U and W (in physical position in space) to represent the images in a matrix (U, W, Z). Thus, the convolution of the reconstructed image Mt or reconstructed image Mt being processed by the PSF may be carried out from any of the representation matrices chosen by the person skilled in the art.

This table represents elementary objects according to preferably a resolution equal to or greater than that of the union of all the photosites of all the colors of a considered optical sensor. It can have a scanning step and positions offset relative to the photosites of the considered sensor, but preferably in spatial agreement with the latter. For example, for an elementary color filter matrix (or "Color Filter Array" in English or CFA) of 4 photosites R (Red), VI (Green 1), V2 (Green 2), B (Blue), arranged on 2 x 2 rows and columns, it can have a digital representation, for each of the for example three color properties R (red), V (green), B (blue), of a pixel opposite the R, then of a pixel opposite the VI, the V2, the B, that is to say in total 4 pixels comprising a channel R, V, B, in spatial correspondence of the 4 photosites R, VI, V2, B of detection, and repeated as many times as necessary in the U and W directions to describe the entire optical sensor considered associated with a given image. Each source image I comprises the intensities of the photosites R (Red), G (Green), B (Blue) (or according to another possible choice of color matrix such as R (Red), Y (Yellow), W (White or White), B (Blue)) of the optical sensor associated with this source image I. As known to those skilled in the art, the source image I may, in addition, comprise other information on the gain and black level of the photosites, photosite by photosite (or region by region where appropriate), i.e. the intensities detected by the photosites will preferably have been corrected to have the same digital level of black restitution, and a level of restitution of a homogenized reference white, for example uniform, or according to a reference in accordance with the sequence of point spread correction treatments. In the event of non-linearity or saturation, the method may comprise a step of correcting this source image I.

It would also be possible to downsample, although this would not result in an improvement in image quality in terms of sharpness, but would allow good quality of a compressed image (in terms of number of pixels).

This table can also be completed, or replaced by a list of objects describing the image. Each object can thus be the square or rectangle pixel, or objects with more properties such as an outline, average color components, but also brightness and color gradient components for example, and a particular shape. This second representation allows more freedom of representation of positions, with possibly coordinates represented by numbers having more resolution than a collection of pixels at predefined locations. In addition, one can authorize the addition of colors and intensities of objects that would be superimposed at certain locations.

The processing phase is now described, i.e. the iterative modification of the reconstructed image M (i.e. the iterations of the processing loop 3 shown in FIGURE 3). Information is also generated from relatively precise knowledge of the optical focusing defects of the imaging system 2 or of one of the optical elements of the imaging system 2 (for example the lens). This information may include the reading of a table providing access to specific information related to the defects of the imaging system 2 or of one of the optical elements of the imaging system 2 (for example the lens), as well as the distance between each lens or system and the scene observed through this lens or system, either considered according to a single distance, or differentiated into object-by-object distance of the scene, and the distance between the lens or system and the sensor associated with this lens or system, and possibly variations in angle between these elements. From this information, a restitution of the point spread functions (called PSF) is synthesized, which can be materialized by coefficient matrices capable of producing by convolution with the intensities ideally emitted by each point of the scene, the intensities obtained at the optical sensor, for each color channel. In the case where displacement defects are known, for example by a representation of the residual movement of non-compensation of the camera module relative to the scene, the PSF matrices are summed at different positions weighted by the time that has elapsed there, to calculate a composite PSF including these residual uncompensated movements, and to use this form of the PSF for the processing phase. To obtain the movement trajectory of a sensor or system imaging 2, it is sufficient for example that an acceleration sensor records the movements of the device containing the camera module 2 during image capture, and if the image is mechanically stabilized with respect to the movements of the device comprising the camera module 2, it would be sufficient for the absolute non-stabilization residual to be calculated with respect to the acceleration sensor and the stabilization signal. In another implementation, in the absence of direct movement measurement data, the image is analyzed in order to determine the result of the movement trajectory and to deduce therefrom the composition to be applied to the PSFs in order to obtain compensation for this movement. In this way, the restored image C will also be compensated for the blurring effects linked to the uncompensated residual movements.

Thus, the processing phase makes it possible to obtain compensation for optical projection defects on the optical sensor, and provides a reduction in noise from detection of light intensities. In particular, it also compensates for mosaic effects with spatially separated detection of several colors, and can also effectively compensate for movement residues of a camera module relative to the scene. The processing phase does not consist of processing the image in successive stages but allows a single processing comprising all the restitution operations.

Also, considering that the processing phase P implemented from a reconstructed image M containing depth information, it is appropriate to calculate the PSFs as a function of the distances to the different objects on the scene, in order to obtain compensation for these optical aberrations taking into account the real focusing distance of each photographed object, which will increase the effective depth of field when restoring the image.

Thus, the processing phase provides as output a so-called restored image C comprising an absence of mosaicking, a significant reduction in noise according to a setting to be adjusted if necessary, a correction of position distortion and blurring of the image adapted to each region of the image, a compensation of relative movements between the lens or the imaging system 2 and the observed scene. The processing phase P therefore does not include a succession of correction steps dissociated from each other. Thus, the effects of conjugations and/or additions of the defects inherent in each step are avoided. It also makes it possible to limit the interpolations of values which would have had to be detected in the absent photosites (because at the location of one of the other detected colors). First of all, it is worth noting that, generally speaking, it is advantageous, but not necessary and in some cases to be avoided, to implement the steps of the method (whether, for example, in the image processing or image reconstruction phase or for any step, in particular of image comparison or comprising an image comparison) from blocks of images rather than on all the pixels considered individually. A block of an image corresponds to an object (or a part of an object) of the scene for which one or more data of the pixels (among the pixel table of the image) constituting the block (representative of the object of the scene) of the image has an identical, similar or equivalent value. Also, it is advantageous to consider as an image block a set of pixels for which one or more of the data that are contained in each of the pixels of the set of pixels constituting the block (and in particular the depth and/or speed information) have identical, neighboring, similar, close or similar values. Such regionalization of the image makes it possible to considerably reduce the time of the calculations carried out on the basis of these images. Furthermore, when the processing or the step implemented from the image comprises or consists of moving (or translating) the position (in the x, y space) of all or part of the pixels within the pixel grid (which is for example the case during the processing phase, during the reconstruction step or during spreading), no loss of resolution is generated in this case.

It should be noted that the steps of the method could be implemented, in part, from image blocks but also, in part, from (isolated) pixel(s) which could not have been grouped in the form of blocks. The term "block" extends in the broad sense, it can, for example, be a group of pixels forming the outline (or part of the outline) of another block. Also, in most cases, it will be possible to constitute images formed from a set of blocks.

It is also possible to homogenize the data(s) having similar or equivalent values by assigning to all the pixels a single and same value for this or these data(s) to the detriment of the image resolution. In other words, it is possible to give an identical value to one or more data contained in the pixels forming a block of the image, when these values have been identified as neighboring, similar, close or similar. The person skilled in the art will know, depending on the intended application, the required image quality and the resources to be allocated to the processing (or the maximum tolerated processing time), adapted a threshold difference from which the data values are no longer considered equivalent.

The processing phase P must take into account the entire model of optical defects by calculating, for each iteration k of the processing loop 3 and for a given imaging system 2 (from which the source image It associated with the reconstructed image Mt is acquired), from the reconstructed image Mt, the optical defects induced on each pixel or group of pixels (or blocks) of a source image It (represented by the values of the different color components of the photosites).

D is determined by comparison of:

- the reconstructed image Mt convolved with the PSF then spread, at least locally, according to a velocity field and/or according to an acquisition duration of the source image It, to which the reconstructed image Bt is associated, with

- the source image It, to which the reconstructed image Bt is associated.

The term D can include the sum of several terms Di. Obtaining the distances Di obtained by comparing the reconstructed image Mt convolved by the PSFs with the source image It, can be, among other things, can be achieved by an evaluation of the local colors in the pixels or groups of pixels in front of the discretization grid of the PSFs, so as to know the colors calculated at the positions of the photosites of the reconstructed image Mt to make the distance comparisons at the appropriate locations, without offset, allows maximum restitution precision.

The optical defect model is taken into account by a calculation, at each iteration k, of the effects that it induces from the reconstructed image M, by composing this image M during processing with the optical defect model, according to the embodiment by a convolution product, noted MCt, between the matrix of the luminous values of the reconstructed image M (at each point of a fairly fine grid), and the PSF matrix on this same grid, which at the output provides a matrix of values MCt according to a grid (grid which allows the comparison with the photosites of the optical sensor from which the source image I was obtained).

Of course, a convolution calculation can be done by multiplication in the frequency domain, and it is also possible to take into account the point spreading effects by the spatial Fourier transforms of the PSF matrices, generally called OTF (Optical Transfer Function), which then simply need to be multiplied by the Fourier transforms of the reconstructed image being restored, then returned to the spatial domain by inverse transform, if this operating mode presents a interest. Point spreading effects can be taken into account by using the TG operator which can be the identity, or a geometric transformation, which consists of locally moving the elements, either by translation of pixels in such a representation, or by equivalent modification of the PSF by recalculation of the set of coefficients to take into account this transformation, or possibly a mixture of the two.

Knowledge of the velocity field within the source images I and/or the reconstructed images M makes it possible to predict and anticipate the movement of objects of a source image It and/or, respectively, of a reconstructed image Mt associated with a successive source image It and/or, respectively, with a successive reconstructed image Mt+1. Thus, the point spreading effects can be applied to the reconstructed image M, prior to comparison with the source image I, using the velocity field. This makes it possible, among other things, to eliminate the blur linked to the aperture time of the imaging system, when taking each image and offers a precise and more faithful restitution of the image of the objects present on the imaged scene (in particular when they are moving in the imaged scene). In addition to predicting or anticipating the movement of objects of a source image It and/or, respectively, of a reconstructed image Mt associated with a successive source image It and/or, respectively, with a successive reconstructed image Mt+1, the velocity field also makes it possible to calculate and predict the future position of objects in motion (or not) in the imaged scene and therefore to know the depth information associated with the objects of the imaged scene. Also, it is possible by analyzing, for example by comparing the position of identical objects within successive images, to know the position of objects occluded in the source image It.

According to a non-limiting embodiment, it is possible to express the result, denoted MCt, of the convolution product of the reconstructed image Mt being processed at iteration k by the PSF, and possibly processed by the geometric transformation TG:

[Math. 4]

MCt = reconstructed image Mt (*) PSF, or

MCt = TG(reconstructed image Mt) (*) PSF, or

MCt = TG(reconstructed image Mt (*) PSF), where (*) means convolution product.

From this comparison, a first indicator or term called distance Di, possibly completed with other terms Di, resulting from a comparison between the image reconstructed Mt convolved by the PSFs (possibly taking into account point spread effects) with the source image It, which we denote by Di(Bt, It), is obtained:

[Math. 5]

D= Dl(Bt, It) + D2 + ... + Dnd.

The term A, called penalty, representative of defects or anomalies within the reconstructed image Mdt, Mgt, is determined from the reconstructed image Mdt, Mgt, is calculated concomitantly with the term D. The second term A comprises: at least one component Ai whose effect is minimized for small differences in intensity between neighboring pixels of the reconstructed image M being processed at iteration k, and/or at least one component A3 whose effect is minimized for small differences in hue between neighboring pixels of the reconstructed image M being processed at iteration k, and/or at least one component A2 whose effect is minimized for low frequencies of changes in direction between neighboring pixels of reconstructed image M being processed drawing an outline.

The second term A can thus include a sum of several terms Ai.

As for the term D, for each iteration k and for a given imaging system (from which the source image It associated with the reconstructed image Mt is acquired), one or more penalties called Pi, P2, ..., P _np , are calculated directly from the reconstructed image Mt or from the reconstructed image Mt being processed.

où nd est le nombre de distances Di, np le nombre de pénalités Pi, avec par exemple : nd = l, np = l, ai = 1 et 0i qui peut être ajusté pour donner plus ou moins de poids entre la distance qui force à se conformer à l'image source I, et la pénalité qui force à rendre l'image reconstruite M, à chaque itération, conforme à une autre propriété, ou à posséder une certaine cohérence par rapport aux images que le système d'imagerie est censé observer. Autrement dit, cet indicateur de pénalité permet aussi de sélectionner les images qu'il est le plus probable d'observer parmi celles qui seront restituées. Par exemple, il est peu probable d'observer une image reconstruite M comportant de la « neige » rouge, vert, bleu à l'intérieur de ses textures, et l'on peut dire que l'indicateur de pénalité P est construit pour rendre moins probable l'apparition de telles images restituées C, quand il est construit pour être sensible à un bruit de couleur dans l'image reconstruite M. La fonction de pénalités Pi va guider vers une solution privilégiée par exemple en empêchant les pixels voisins de présenter des valeurs ayant un grand écart entre elles (hormis, par exemple, pour les pixels voisins blocs/contours) et ainsi réduire le bruit de l'image restituée C par rapport à l'image source I potentiellement bruitée ou encore de limiter les effets de créneaux sur les contours des blocs ou objets de l'image restituée C. La fonction de pénalités Pi va, entre autres, va également améliorer l'information de profondeur et de vitesse en empêchant les pixels voisins de présenter des valeurs trop différentes, de supprimer les valeurs aberrantes au sein d'un même bloc. A global deviation indicator E is obtained which, according to the non-limiting embodiment, can be expressed as: [Math. 6]

where nd is the number of distances Di, np the number of penalties Pi, with for example: nd = l, np = l, ai = 1 and 0i which can be adjusted to give more or less weight between the distance which forces to conform to the source image I, and the penalty which forces to make the reconstructed image M, at each iteration, conform to another property, or to have a certain coherence with respect to the images that the imaging system is supposed to observe. In other words, this penalty indicator also makes it possible to select the images which are most likely to be observed among those which will be restored. By example, it is unlikely to observe a reconstructed image M with red, green, blue "snow" inside its textures, and we can say that the penalty indicator P is built to make the appearance of such restored images C less likely, when it is built to be sensitive to color noise in the reconstructed image M. The penalty function Pi will guide towards a preferred solution for example by preventing neighboring pixels from having values with a large gap between them (except, for example, for neighboring block/contour pixels) and thus reduce the noise of the restored image C compared to the potentially noisy source image I or even limit the effects of slots on the contours of the blocks or objects of the restored image C. The penalty function Pi will, among other things, also improve the depth and speed information by preventing neighboring pixels from having values that are too different, by removing outliers within the same block.

In particular, according to the embodiment, the function E comprises or consists of the sum of the first term D and the second term A. Also, at the first iteration k=0, the processing loop at the processing phase PO compares the source image 10 and the result MCt of the convolution product of the reconstructed image MO at iteration 0 (i.e. unmodified or as resulting from the reconstruction step) by the PSF, and possibly processed by the geometric transformation TG so as to take into account a maximum of elements precisely modeling the geometry of all the parts of the optical sensor, the optical focusing effects throughout the sensor, any movements, the distance effects modifying the focusing effects.

The implementation of the method aims to obtain a restored image C which minimizes the indicator E.

Except in special cases where the expression of the distance Di would allow a direct calculation to minimize it, and where the penalty PI would also allow it, or would be taken as zero, the general method of resolution can for example be obtained by a method called Newton's gradient, which allows the minimum of a function to be found, by varying the parameters or input data inversely to the gradient of a function E, to minimize its value.

Thus, at each iteration, we calculate the sum of the derivatives of each term of E with respect to all the parameters or data of the reconstructed image Mt-1 at the previous iteration k-1, then we evolve the elements of the reconstructed image Mt at iteration k with respect to this gradient. The aim of the iterations is to modify the image reconstructed M at each step, so as to minimize the indicator E. The parameters or data of the image, and therefore of E, are, depending on the embodiment, the values R, G, B and the depth information (Z) and/or the speed information (V). It may also be advantageous to consider the position (v, w) of any pixel or any image block (or the physical coordinates (x, y)) as a parameter.

Although the particular embodiment described is based on the use of partial derivatives of the function E with respect to the reconstructed image M being processed to minimize the function E, any other techniques for minimizing the function E are applicable. With regard to the non-limiting embodiment described, it is possible to calculate a local matrix of partial derivatives dE/di( ,w') where i(v,w) is one of the data or a parameter contained in a pixel or a block, at the coordinates (v,w), of the reconstructed image M being processed. Also, the minimization of E will amount, depending on the embodiment, to calculating dE to find its optimal value. dE can be expressed as the sum of the partial derivatives dE/di(y,w).

As detailed above, it is preferable to implement the method on image blocks. In addition to the R, G, B, V and Z parameters, it may be advantageous to use other partial derivatives with respect to other parameters such as, for example, a block dilation parameter and/or a block contour parameter representative of the dilation and contour effects of objects in the scene (contour which may, in some cases, be considered as a block). It may also be advantageous to consider the position of any pixel or any image block as a parameter, which would add to dE the partial derivatives dE)dv and dE/dw.

Also, depending on the embodiment, dE can be written:

[Math ^dE = ^d

[Math dE = ^d

[Math dE = (

Finally, it is recalled that this minimization of E is carried out at each iteration k of each processing phase Pt. Also, the derivative dE, as well as each of the partial derivatives dE/di(y,w) can be indexed by the iteration k at which it is calculated as well as the time t at which the source image It associated with the current processing phase t was acquired d(E(R,G,B,Z,V,v,w)k,t) and dE/di(y,w') _kt . The description of the embodiment using partial derivatives for the minimization of the function E is transposed, directly, to the difference term D. Also, the minimization of E will tend to ensure that each of the terms Di = dD/di( ,w) _kt and Pi = dP/di( ,w _kt is minimized, that is to say to reduce the disparities between the different parameters of the reconstructed image Mt being processed at iteration k with respect to the source image It and guide the restitution of the image towards what is expected.

The convolution of the reconstructed image M by the PSF will contribute to reducing the distance term Di. As such, everything described in the application regarding PSFs, discretization of the PSF grid, geometric transformation (GT) (of the image before convolution or of the convolution product) obviously applies here.

In order to further improve the image quality, and in particular the restitution of the depth and speed parameters, it is possible to implement the step of updating the depth and speed information during the processing phase, i.e. between two iterations k of the processing phase. In this case, the update step is performed from the reconstructed image Mgt being processed in the iterative loop associated with the source image Igt and the reconstructed image Mdt being processed in the iterative loop associated with the source image Idt. A geometric transformation (GT) may sometimes be necessary before the comparison. This step is described in detail below.

In addition, preferably, contour search or extraction operations are to be carried out on the reconstructed image M or reconstructed M currently being processed at at least certain iterations, in order to create, move elements of the reconstructed image M or reconstructed M currently being processed.

The distance D or the distances Di therefore come from the reconstructed convolved images Mt (*) PSF (or TG (reconstructed image Mt (*) PSF) while P or the Pi come directly from the reconstructed images Mt.

At the end of a processing phase Pt, either the value of E is judged sufficiently small, or an estimate of the gradients relative to it is sufficiently small, or the number of iterations is limited to estimate that the image obtained is restorable.

The calculations of the distances Di are preferably made on the set of positions of the photosites. It is therefore appropriate to calculate the values of the different intensities of the colors from [Math. 4]. The Pi penalties can be calculated at positions not necessarily related to the positions of the photosites. They can be calculated from pixels or groups of pixels of the reconstructed image Mt-1 at the previous iteration k-1, or from properties of larger objects, without necessarily showing the color properties according to the precise image restitution grid or the PSF for example, but by a more global calculation from properties of the object.

In general, and for each imaging system considered, the function describing the response of the at least one imaging system, the PSF according to the non-limiting embodiment, may depend on:

- a distance Z between the optical sensor and an object in the scene imaged by the optical sensor, and/or

- a relative position of the imaging system considered with respect to at least one other imaging system, and/or

- a distance between a part of the imaging system and the optical sensor of the imaging system, and/or

- a distance between a part of the imaging system and an object in the scene imaged by the optical sensor, and/or

- a state of the imaging system, such as a zoom or focus or numerical aperture setting of the imaging system, and/or

- of the pixel of the image being reconstructed M or of the reconstructed image M being processed and/or of the restored image C and/or of the photosite of the sensor, and/or

- of the information contained in each pixel or group or block of pixels of the convolved image, and/or

- of each position in the image field, in particular according to the depth of the object, and/or

- of each pixel or group or block of pixels, in particular depending on the depth information contained in each pixel or group of pixels), and/or

- one or more angles between the optical sensor and the imaging system.

One of the objectives of the invention being to improve and make reliable the depth information contained in the enriched images, it is important that the PSF is a function, among other things, of the depth field so that the restored image C is sharp, has a maximum effective depth of field and accurately reflects the distance between the objects in the scene and the lens and/or the sensor. Thus, for an image reconstructed M given, there will therefore be at least as many PSFs as there are pixels or image block(s) comprising different depth information.

The distance between the imaging systems 2 must be known. Preferably, the imaging systems 2 are stationary relative to each other. The orientation, relative and/or relative to each other, of the sensor and/or the lens and/or the imaging systems 2 to each other is known. In the case of a smartphone or a car, all these parameters are known and do not change.

The distance Z between the optical sensor and an object of the scene imaged by the optical sensor is known or, preferably, determined or calculated according to the invention. Also, the depth information determined or calculated can be used to feed the PSF during the successive implementations of the steps of the method according to the invention.

Although the processing phase P may be sufficient to obtain an improved image C, in particular comprising solidified depth and speed information. The reconstruction step is not necessarily implemented before or during each processing phase. It is even possible that the method does not comprise a reconstruction step. It is possible, for example, to assign a predetermined or default depth and/or speed information value to the pixels of the source image It. The processing phase P described above will make it possible to improve the depth and/or speed information as described above. It is even possible to enrich the source image It with several depth and/or speed information values, certain blocks being, for example, assigned a depth and/or speed value different from certain other blocks. However, to further improve the quality of the restored images C, it is preferable for the method to comprise a reconstruction step.

With reference to FIGURES 3 to 5, the method 1 comprises, prior to and/or, respectively, during the processing phase, a step of reconstructing the reconstructed image Mdt, Mgt or respectively the reconstructed image Mdt, Mgt currently being processed from at least one source image I and/or at least one reconstructed image M and/or at least one reconstructed image M currently being processed and/or at least one restored image C. Also, at least a portion of the pixels of the reconstructed image Mdt, Mgt comprises depth information. A reconstructed image is understood to mean an image of which at least a portion of the pixels comprises depth information. In other words, apart from the source images I, all of the Process images are enriched with depth information, in particular is enriched with a depth field.

The method is of particular interest when the imaging systems, the imaged scene and/or objects of the imaged scene are in relative motion with respect to one (one) other (the others). Indeed, in this case, it is advantageous for the reconstruction step to comprise an extraction of the depth information from the imaged scene. However, the depth information may not be extracted or may not be extracted prior to or during each processing phase p. Indeed, by way of non-limiting example, the reconstruction step may be implemented from depth information not extracted but obtained by or originating from, for example, a depth information measurement system such as, for example in the case of an autonomous vehicle, a LIDAR. This depth information can be added and/or substituted in at least part of the pixels of an image (source I, reconstructed M, reconstructed M during processing or restored C) in order to obtain a reconstructed image Mdt, Mgt. Still by way of non-limiting example, the reconstruction step, i.e. obtaining a reconstructed image Mdt, Mgt, can comprise or consist of a displacement of image blocks, for example from the restored image Cdt-1, Cgt-1 to the previous processing phase Pt-1, according to a speed field and/or a depth field (preferably, but not necessarily, contained in the pixel table representing the image).

For this purpose, the method may also comprise an extraction of speed data and/or the initialization step may comprise an addition and/or a substitution of speed data from at least two source images It, It-1, It-2..., from at least two reconstructed images Mt, Mt-1, Mt-2..., from at least two reconstructed images Mt, Mt-1, Mt-2... currently being processed, from at least two restored images Ct, Ct-1, Ct-2..., from at least one source image It-1, It-2, It-3... and at least one resituated image Ct, Ct-1, Ct-2..., from at least one reconstructed image Mt-1, Mt-2, Mt-3 and at least one restored image Ct, Ct-1, Ct-2... or from at least one source image It-1, It-2, It-3... and at least one reconstructed image Mt, Mt-1, Mt-2. In the same way, the method can also comprise a step of updating the speed information contained in an image from at least two source images It, It-1, It-2..., from at least two reconstructed images Mt, Mt-1, Mt-2..., from at least two reconstructed images Mt, Mt-1, Mt-2... currently being processed, from at least two restored images Mt, Mt-1, Mt-2..., from at least one source image It-1, It-2, It-3... and at least one relocated image Ct, Ct-1, Ct-2..., from at least one reconstructed image Mt-1, Mt-2, Mt-3 and at least one image restored Ct, Ct-1, Ct-2... or at least one source image It-1, It-2, It-3... and at least one reconstructed image Mt, Mt-1, Mt-2 but also from reconstructed images Mt currently being processed at previous iterations k-1, k-2, k-3... of the iterative loop 3 of the current processing phase Pt. The extraction of speed data or information is carried out from two (or more) distinct images, preferably, but not necessarily, acquired by the same imaging system 2, at two distinct times so that the moving objects in the imaged scene are in two distinct positions on the two (or more) compared images.

According to a non-limiting embodiment, the extraction of the velocity field comprises the comparison of two or more images originating from the same imaging system 2 (r or g) and acquired at different times. A step of identifying a pixel or a group or a block of pixels common to the two (or more) compared images is carried out. This step aims to identify the same object which is present on the two compared images. The comparison relates to any information or value or data contained in the pixels of the pixel matrix of the compared images. Preferably but not limitingly, the information can be color data and/or texture data, and/or brightness data, and/or hue data, and/or saturation data, and/or RGB data.

When the objects common to the two images have been identified, the speed information is extracted from the positions (and/or the difference in position) of the pixel or group or block of pixels identified on the two compared images and the time elapsed between the two images (time lapse between t and t-1 for example if the two compared images are It and It-1). It should be noted that an extraction from the restored images (for example Rt and Rt-1) is advantageous because of the better quality, the better resolution and the few defects that the images restored according to the method include in comparison with the source images I.

Knowledge of the relative movement speed of the imaging systems 2 with respect to the imaged scene, and possibly of the imaging systems 2 with respect to each other, can be taken into account for a better resolution of the extracted velocity field or even can be used to deduce the velocity field which can thus be added and/or substituted in the pixel table of the image. Therefore, the velocity field is not necessarily determined by extraction but can be obtained by or come from, for example, a depth information measurement system such as, for example in the case of an autonomous vehicle, a rev counter, a tachometer or a LIDAR. Thus, according to the invention, the images (reconstructed Mt, reconstructed Mt during processing and restored Ct) can also be enriched with a velocity field. Consequently, the velocity field contained in the images (reconstructed Mt-1, Mt-2, Mt-3..., reconstructed Mt-1, Mt-2, Mt-3... during processing and restored Ct-1, Ct-2, Ct-3...) coming from previous processing phases Pt-1, Pt-2, Pt-3 can be used for implementing the extraction step, the initialization step (including moving image blocks according to the velocity field) and the velocity data updating step.

Preferably, for one, several or each of the processing phases P, the method 1 preferably comprises a reconstruction step implemented prior to the or each of the processing phases P. With reference to FIGURES 3 to 5, the reconstruction step implemented prior to the processing phase P is called the initialization step. Furthermore, alternatively, preferably in a complementary manner, the reconstruction step can be implemented during the processing phase. This is particularly advantageous when a large disparity between the source images Id and Ig and/or the reconstructed images Md and Mg and/or the restored images Cd and Cg is observed or when the differences D are significant. The reconstruction step implemented during the processing phase P is called the depth information updating step. However, in some cases, for example when few or no significant disparities are observed between the source images Id and Ig and/or the reconstructed images Md and Mg and/or the restored images Cg and Cd or when the differences D are small or negligible, the method may not comprise, for one or more processing phases P, a reconstruction step. In this case, the restored image Ct-1 at the processing phase Pt-1 may be used as the reconstructed image Bt at the processing phase Pt, possibly with the implementation of block displacements of the image Ct-1 (as described in detail above and below) prior to the processing phase Pt.

For a processing phase Pt considered, the reconstruction step therefore allows obtaining an image (reconstructed Mt, reconstructed Mt during processing or restored Ct) in which depth information has been added and/or substituted in at least part of the pixels.

Furthermore, as detailed above, the efficiency of the processing phase makes it possible to reduce the resources required for processing the source images Idt, Igt and allows therefore to process the source images Idt, Igt in real time and to restore images Cdt, Cgt prior to the acquisition of the source images Idt+1, Igt+1. Thus, with reference to FIGURE 1, the image Ct, associated with the source image It acquired at time t, which is restored at the end of the iteration Kl, is generally and ideally obtained before the acquisition of the source image Ct+1. Thus, the restored image Ct can be used for the processing phase Pt+1, and if necessary for the subsequent processing phases Pt+2, Pt+3.... The reconstructed image Mdt or reconstructed Mdt currently being processed can be used for the reconstruction of a reconstructed image Mdt+1, Mdt+2 associated with another source image (preferably subsequent) Idt+1, Idt+2 or for the reconstruction of a reconstructed image Mgt, Mgt+1, Mgt+2 associated with a source image Igt, Igt+1, Igt+2 originating from another imaging system. The restored image Cdt can be used for the reconstruction of a reconstructed image Mdt+1, Mdt+2 associated with another source image (preferably subsequent) Idt+1, Idt+2 or for the reconstruction of a reconstructed image Mgt, Mgt+1, Mgt+2 associated with a source image Igt, Igt+1, Igt+2 originating from another imaging system. This is of particular interest because the processing phase makes it possible to obtain reconstructed images M during processing and restored images C whose quality is much higher than that of the source images I. Also, the reconstruction step is preferably implemented from reconstructed images Mt or reconstructed images Mt during processing or, in a particularly advantageous manner, from restored images Rt-1 at the previous processing phase, and/or at the previous processing phases. In this way, since the depth information is extracted from higher quality images (containing few or no defects, or at least fewer defects than the source images), the depth information contained in the rendered images C is more reliable, more precise and includes fewer errors.

The initialization step and/or the depth information updating step can be implemented by extracting the depth information from at least two images (source I, reconstructed B, reconstructed B being processed and/or restored R). With reference to FIGURE 6, a method for extracting the depth information known to those skilled in the art under the name of photogrammetric comparison is illustrated. This method is well known to those skilled in the art and will not be described in detail. This method comprises comparing two or more images from separate imaging systems 2 (two or more) placed in two separate positions in space. A step of identifying a pixel or a group or block of pixels common to the two (or more) compared images is performed. This step aims to identify the same object that is present in both compared images. The comparison relates to any information or value or data contained in the pixels of the pixel matrix of the compared images. Preferably but not limited to, the information may be color data, and/or texture data, and/or brightness data, and/or hue data, and/or saturation data, and/or RGB data.

When the objects common to the two images have been identified, the disparity is measured or determined for each common object, including the offset (in coordinates or pixel positions) of the common object of one image relative to the other. In practice, the depth information is extracted from the disparities in accordance with FIGURE 6 using principles of optics and geometry available to those skilled in the art. It should be noted that the embodiment presented is only a non-limiting example. Those skilled in the art will be able to adapt this method or choose another of the existing methods.

Unlike the extraction of the velocity field which is carried out, preferably from two images of the same scene acquired at two different t, t-1, the extraction of the depth field is carried out, preferably, from two different images, acquired at the same time t, coming from the same scene. However, it should be noted that in both cases, it is possible that objects are occluded by certain other objects of the scene so that the velocity and/or depth field associated with these objects cannot be extracted. Therefore, the displacement of image blocks, according to the velocity and/or depth field (as described in detail above and below) makes it possible to solve this problem.

In the case of an autonomous vehicle, the extraction of depth information makes it possible, among other things, not to equip the autonomous vehicle with a LIDAR. In this case, the extraction of depth information can be carried out by comparing at least two source images I. During the initialization step, the comparison can advantageously be carried out from the two source images Idt, Igt acquired at time t. In the case where no depth information is provided by a depth information measurement system, the initialization step by comparing the two source images IdtO, IgtO acquired at time tO is necessary so that the reconstructed images MdtO, MgtO include depth information prior to the implementation of the processing phase PO. The extracted depth information is entered in all or part of the pixels of the pixel matrix source images IdO, IgO used during the comparison to obtain the reconstructed images MdO, MgO. It should be noted that the extraction can be carried out by comparing as many source images I acquired at the same time t as long as they include or correspond to, at least in part, the same scene.

The extraction of depth information by comparison of at least two source images It acquired at time t, can be implemented, for any initialization step and for any depth information update step (and not only for the initialization step nor only for the source images 10 acquired at time t = 0). In addition, for any source image It (whatever the acquisition time t), if the pitch of the reconstructed image Mt differs from the pitch of the associated source image It, for example due to geometric transformation calculations, it is then appropriate to interpolate the values of the pixels in the frame of reference of the reconstructed image Mt, to obtain a representation matrix at positions compatible with the matrix for example of PSF so that the output coefficients of the convolution Reconstructed image B (*) PSF correspond to the positions of the photosites, without offset. In the absence of this precaution, the quality of the reconstructed image Mt, and therefore the restored image Ct, may include defects such as shifts between the restored colors, causing for example the appearance of colored fringe residues around the elements of the restored image, or geometric micro-distortions linked to these fractional parts in the calculated position indices not taken into account.

With reference to FIGURES 3 to 5, and for the processing phases P where t>0, the extraction of the depth information, used for the implementation of the reconstruction step, can also be carried out from the comparison:

- at least one source image, for example: Idt and Igt, or at least two reconstructed images, for example: Mdt and Mdt-1 and/or Mgt and Mgt-1 and/or Mdt and Mgt, or at least two reconstructed images M currently being processed, for example: Mdt and Mdt-1 and/or Mgt and Mgt-1 and/or Mdt and Mgt, or at least two restored images, for example: Cdt-1 and Cdt-2 and/or Cgt-1 and Cgt-2 and/or Cdt-1 and Cgt-1, or at least one source image (Idt or Igt) with at least one restored image M, preferably at a previous processing phase (for example Mdt-1 or Mgt- 1) (or at previous processing phases Mdt-2, Mdt-3... or Mgt-2, Mgt- 3...), or of at least one source image (Idt or Igt) with at least one reconstructed image (Mdt or Mgt) and/or with at least one reconstructed image (Mdt or Mgt) being processed, or of at least one source image (Idt or Igt) with at least one restored image C, preferably at a previous processing phase (e.g. Cdt-1 or Cgt-1) (or at previous processing phases Cdt-2, Cdt-3... or Cgt-2, Cgt-3...), or of at least one reconstructed image (Mdt or Mgt) and/or of at least one reconstructed image (Mdt or Mgt) being processed with at least one restored image C, preferably at a previous processing phase (e.g. Cdt-1 or Cgt-1) (or at previous processing phases Cdt-2, Cdt-3... or Cgt-2, Cgt-3...), or of at least one a reconstructed image (Mdt or Mgt) being processed with at least one reconstructed image (Mdt or Mgt).

Preferably, when the extraction is performed by comparing a source image I with a reconstructed image M, a reconstructed image M being processed or a restored image C, the source image I is convolved by an inverse function of the PSF, called iPSF, prior to its comparison with the reconstructed image M, the reconstructed image M being processed or the restored image C. This makes it possible to obtain a better correlation between the source image I and the image to which it is compared. The iPSF is chosen so that the composition of the PSF with the iPSF tends to give the identity function at least on the low frequencies of the image (which have overall a good signal-to-noise ratio). The inversion of the PSF is arbitrary because it is not possible to invert (generally amplify) all the frequency bands, only the bands having a good signal-to-noise ratio can be amplified to avoid degrading the image. This is the concept known in the literature, of Wiener filtering: typically, the spatial Fourier transform (in dimension 2) of the PSF is calculated. Preferably, an amplification of the low frequencies is applied up to a certain threshold (typically a frequency that can be previously determined so as not to degrade the image) in such a way as to make a PSF gain composed with flat iPSF up to the threshold. Preferably again, the amplitude of the higher frequencies is voluntarily limited to limit the appearance of noise and high-frequency artifacts in the image. Preferably, from the PSFs, a certain inversion of the matrix representing it is calculated, at least on the lowest, significant spatial frequencies. From this matrix called iPSF, which can be preferentially specific according to each wavelength, and each position in the image field, and one or more pieces of information or data or values contained in a pixel or in one or each group or block of pixels, in particular according to the depth of the object of each object, a convolution product is carried out with the values of the neighboring photosites from the source image I to produce a first reconstructed image value M or of the reconstructed image M being processed to initialize the restitution table.

Prior to comparing images with each other (source(s) I, reconstructed M, reconstructed M being processed and/or restored C), it may be advantageous to use an entity called geometric transformation (GT) providing a geometric transformation channel of each scene to make it spatially coincide between the different images. This TG is generally not linear to take into account differences in distortions between the imaging systems 2 and that the modeling of the geometric distortions is not perfect, or that the scene has moved relative to one or more imaging systems. For example, the TG of each source image, for example IoT, IgO acquired at t=0, can be calculated for example by seeking correlation optima by moving parts of an IoT source image relative to another IgO source image. The translation that made it possible to obtain the best correlation is retained for subsequent image comparisons (at t>0). This operation is to be repeated on the set of several parts of each image, not necessarily exhaustively on all parts of the image. Preferably, the translation field obtained is interpolated by an interpolation method, for example linear, or preferably cubic, or any other method that allows such interpolation. It is the set of these interpolations that makes it possible to obtain complete coverage of the image field to obtain each TG operator.

As mentioned above, for processing phases P>PO where t>0, and for a processing phase P considered, the reconstruction step is carried out, without extracting depth information from images, by moving image blocks, according to a velocity field and/or a depth field:

- of the restored image Ct-1 obtained in the processing phase Pt-1, in particular to obtain a reconstructed image Mt (initialization step), or - from the reconstructed image Mt to the processing phase Pt, in particular to obtain the reconstructed image Mt currently being processed (step of updating the depth information prior to one or more of the iterations k>0 of the processing phase Pt), or

- from the reconstructed image Mt-1 to the processing phase Pt-1, in particular to obtain the restored image Ct-1 (step of updating the depth information carried out during the last iteration k=K2 of the processing phase Pt), or

- of the source image It, in particular to obtain a reconstructed image Mt (initialization step).

The initialization step, by extracting depth information from at least two source images Idt, Igt, can be implemented once only prior to the first processing phase PO. The subsequent reconstruction steps, implemented beforehand or during the subsequent processing phases (P>PO at t>0) can be implemented without depth extraction but, for example:

- by moving blocks of an image (source I, reconstructed M, reconstructed M during processing or restored C) according to a speed field and/or a depth field, or

- by adding and/or modifying depth information in at least part of the pixels of an image (source I, reconstructed M, reconstructed M during processing or restored C) obtained by or originating from a depth information measurement system.

Preferably, the method 1 comprises the implementation of at least one reconstruction step, preferably at least one initialization step, by processed source images It, that is to say by processing phases Pt.

The velocity field can also be taken into account during the reconstruction step. The velocity field is preferentially associated with objects identified within an image. Also, in order to extend this field in the parts of the objects where velocity data are missing, the image associated with time t-1 can be moved locally to produce the image expected at time t. For this, each object can be moved according to its own velocities by taking into account the time elapsed between t-1 and t. As an example, the best correlation could be determined around of moving the elements of an image associated with time t-1 to manufacture the initialization of the reconstructed image Mt at time t. The correlation is established, for example, between the source image It convolved by the iPSF and the reconstructed image Mt-1, the reconstructed image Mt-1 being processed or, preferably, the restored image Ct-1 in which pixel or image block movements are carried out according to the velocity field. In other words, this amounts to considering the movements of objects, according to the depth velocity field and for a time interval between t and t-1 elapsed in the reconstructed image Mt-1, the reconstructed image Mt-1 being processed or, preferably, the restored image Ct-1, varying the position of the objects obtained so as to obtain the best correlation. Thus, knowledge of hidden objects, as well as their depth and/or speed, in the source image It is known.

As specified above, the depth field (Z) can also be taken into account during the reconstruction step to calculate and identify hidden objects at time t. Indeed, according to the invention, the images (reconstructed Mt-1, Mt-2, Mt-3..., reconstructed Mt-1, Mt-2, Mt-3... during processing and restored Ct-1, Ct-2, Ct-3...) coming from previous processing phases Pt-1, Pt-2, Pt-3... are enriched with a depth field. Therefore, it is appropriate to move the objects, generally locally, according to the depth field, preferably by ordering them, according to the depth field, during reconstruction. In practice, only the pixels of the pixel matrix corresponding to the objects in the scene closest to the imaging systems, i.e. those whose position variations are the most marked, are moved. Thus, the reconstructed image Mt-1, the reconstructed image Mt-1 being processed or, preferably, the restored image Ct-1 can, at least in part, serve as a basis for the reconstruction step of the reconstructed image Ct. In addition, the depth information contained in the pixel matrix can make it possible to generate a stack of object planes at different depths. Each plane will include the objects present at the depth of the plane considered. In this case, the displacement of objects, according to the depth field, can be carried out in each plane of the stack of planes. The calculations can be simplified in that only the objects included in the planes of the scene closest to the imaging systems 2 are moved. In general, it should be noted that the depth field contained in the images (reconstructed Mt-1, Mt-2, Mt-3..., reconstructed Mt-1, Mt-2, Mt-3... during processing and restored Ct-1, Ct-2, Ct-3...) coming from previous processing phases Pt-1, Pt-2, Pt-3... can be used for the implementation of the step extraction, the initialization step (including moving image blocks according to the depth field) and the update step.

The reconstruction of the reconstructed image M including the consideration of the speed and/or depth field is particularly advantageous because it makes it possible to know the position of hidden objects, as well as their depth in the imaged scene, in the source image It. This information, entered in the pixel matrix without necessarily being restored in the displayed restored image Mt, is crucial in the case of autonomous vehicles because the automatic driving system contains information, not present in and not deducible from the source image It, allowing it, for example, to anticipate the upcoming presence of an obstacle on the trajectory or the detection of a traffic sign or a traffic light that is not visible on the source image It.

In general, all the characteristics relating to the depth field (which concern the enriched images, the enrichment of the images, the reconstruction of the images, the processing phase P of the images, the extraction of data, the initialization step, the step of updating the depth data, etc.) are transposable, mutatis mutandis, to the velocity field and vice versa.

In general, according to the invention, it is advantageous, when extracting data (depth information, velocity field and depth field) and/or during the initialization step (including when moving objects according to the velocity field, or according to the velocity field and the depth field, when reconstructing the reconstructed image Mt at k>0), to use as the starting base(s) or image(s) (for the extraction and/or for the initialization or for the movement of blocks), an image restored Ct-1 in the previous processing phase Pt-1 or an image restored Ct-2, Ct-3 ... in one of the previous processing phases Pt-2, Pt-3 ..., because the latter will already include all the different modifications made during the previous processing phase Pt-1. This can reduce the iterations to obtain the rendered image Ct at the processing phase Pt compared to the composition method with the iPSF for example. In addition, the source image It is generally close to the rendered image Ct-1 at the processing phase Pt-1 which achieves the exit criteria of the iteration loop (processing phase), because few details change between it-1 and it, except the movement of objects. The convergence of the iteration loop can then be all the faster, i.e. include fewer iterations. Thus, we obtain an image closer to the end-of-iteration image k, than when from initialization to it = 0 or during a change of plane. The translation of image parts (according to the velocity field and according to the velocity field and the depth field) is generally less resource-intensive than convolution calculations with PSF. Indeed, they correspond to pixel shifts, and/or interpolations, one-dimensional while the PSF is two-dimensional. Thus this initialization method is particularly advantageous. It generally allows to make fewer iterations k during the processing phase. It also allows to perform calculations only on parts of images to be refined, without having to perform calculations on the entire image and for parts where the image already meets the comparison criteria (as is the case in state-of-the-art image processing methods or a succession of steps performed on the entire image are implemented).

In general, and as detailed above, it is advantageous to extract data and/or implement the initialization step from a rendered image Ct-1 at the previous processing phase Pt-1 or from a rendered image Ct-2, Ct-3 ... at one of the previous processing phases Pt-2, Pt-3.... The previous processing phase Pt-2, Pt-3... to which it is possible to go back to use the rendered image Ct-2, Ct-3... depends, in particular, on the acquisition frequency of the source images I, the speed of movement of the objects in the scene and/or the imaging systems 2 relative to the scene. Furthermore, the time elapsed between the acquisition of the source image It-2, It-3... associated with the previous processing phase Pt-2, Pt-3... up to which it is possible to go back to use the restored image Ct-2, Ct-3... and the acquisition of the source image It corresponding to the current processing phase Pt must not be too high so that the majority of the objects in the scene are still present in the scene. The fact that the objects have moved, even significantly, during this time lapse is not critical in that it is possible to identify identical objects (having different positions) on different images. Furthermore, and in particular for depth and/or speed extraction, it may be interesting to perform multiple comparisons between reconstructed images Ct-1 with Ct-2 and Ct-1 with Ct-3 and Ct-2 with Ct-3... to obtain a trend of variation in depth and/or speed of the objects in the scene and thus better predict and anticipate the position and/or depth of future objects. This also makes it possible to obtain more precise and reliable depth and/or speed information.

Generally speaking, everything detailed above regarding data extraction and/or implementing the initialization step from images restored Ct-1, Ct-2, Ct-3... to previous processing phases Pt-1, Pt-2, Pt-3... is directly transposable to the update step carried out from reconstructed images Mt-1, Mt-2, Mt-3... or from reconstructed images Mt-1, Mt-2, Mt-3 currently being processed at previous processing phases Pt-1, Pt-2, Pt-3... but also from reconstructed images Mt currently being processed at previous iterations k-1, k-2, k-3... of the iterative loop 3 of the current processing phase Pt.

However, there is nothing to prevent the initialization step from comprising the reconstruction of the reconstructed image Mt from the source image It, then convolving the image thus reconstructed Mt by the iPSF, then, possibly, moving pixels or blocks of the image thus reconstructed convolved Mt according to the velocity field or according to the velocity field and the depth field as described above. In particular, it is advantageous that the reconstruction step, and in particular the initialization step, is carried out by extracting the information by comparing the source images Idt, Igt when significant position distortions in the PSF are present, or when focusing aberrations (blur) would be expected, in order to benefit from more precise data in Z prior to the processing phase Pt. Indeed, in this case, the convolution by the iPSF of the source image I, before reconstruction, may be preferable because this makes it possible to take into account position distortions and precorrection of focusing aberrations (correction of the 'blur') which provides more precise and more faithful images of the imaged scene. Consequently, this proves advantageous because the extraction is carried out from more precise images containing fewer errors and therefore allows a better evaluation of the disparity between the compared images and therefore of the depth.

Thus, in summary, and according to the variants of the invention and/or the embodiments of the invention and/or the combination of variants and embodiments, the invention has the following advantages: not producing a reduction in sharpness at a noise reduction step which must then be compensated for, which is difficult to maintain good homogeneity of the compensation over the entire image, and/or

- to avoid the appearance of colored fringes often visible around the contours of objects in the image, and/or - to avoid the appearance of color noise (a sort of red, green, blue snow) inside the texture of objects, or more generally in certain areas of the image, and/or

- to include in the rendered image all corrections to take into account distortion, as well as loss of sharpness, in a uniform and precise manner across the entire image, and/or

- to avoid false color effects in parts of the objects' texture, and/or

- to improve sharpness across the entire depth of field of the image, and this beyond the sharpness zone of the hyperfocal distance, provided that the Z of each object is taken into account in the PSF (X, Y, Z), and/or

- to provide enriched images with a depth field (Z) and/or a velocity field presenting precise depth and/or velocity information, and/or

- to know the position and/or movement and/or depth of hidden objects, i.e. objects that cannot be observed because they are hidden by other objects in the scene, and/or

- to predict or forecast the future movement and/or position of objects in the imaged scene and/or the future depth and/or depth changes of said objects.

Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.

Thus, in variants that can be combined with each other of the embodiments described above: the reconstructed image Mt is obtained by merging at least two reconstructed images, for example Mt-1 and Mt-2 or Mdt-1 and Mgt-1, and/or the processing phase P comprises a step of merging at least two restored images, for example Cdt and Cgt and/or Cdt-1 and Cdt, to provide at least one restored image Ct, and/or there is proposed according to the invention a device, a system or any apparatus comprising a processing unit implementing any one of the embodiments of the method according to the invention that has just been described for correct/process/restore images acquired by one or more imaging systems 2 of said device, said system or said apparatus or to correct/process/restore images stored in said device, said system or said apparatus, said device, said system or said apparatus being able to be, by way of non-limiting examples: a smart mobile phone (or smartphone), a computer, a camera, a vehicle, a drone, a medical device or a satellite, and/or it is proposed according to the invention a device, a system, a machine (preferably motorized) or any apparatus comprising a processing unit implementing any one of the embodiments of the method according to the invention just described to correct/process/restore images acquired by one or more imaging systems 2 of said device, said system, said machine or said apparatus or to correct/process/restore images stored in said device, said system, said machine or said apparatus, said device, said system, said machine or said apparatus being able to be, by way of non-limiting examples: a smart mobile phone (or smartphone), a tablet, a camera, a vehicle, a drone, a medical device or a satellite, and/or it is proposed to use the rendered images Ct, preferably a stream of rendered images Ct, obtained by implementing the method according to the invention from images, preferably from a stream of images, preferably originating from a smartphone, to feed, preferably with a continuous stream of rendered images Ct, a virtual three-dimensional space (for example, the metaverse), and/or to use any of the embodiments of the device according to the invention just described and/or any of the embodiments of the method according to the invention just described within a device, a system, a machine or an apparatus just described or within a processing unit from images, preferably a stream of images, originating from a device, a system, a machine or of an apparatus just described, and/or there is provided according to the invention a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to any one of the embodiments described, and/or according to the invention there is provided a medium readable, in particular, by computer or by any device comprising a processing unit comprising instructions which, when executed by said computer or said device, lead the latter to implement the method according to any one of the embodiments described.

In addition, the various features, forms, variations and embodiments of the invention may be combined with each other in various combinations to the extent that they are not incompatible or mutually exclusive.

Claims

1. Method for restoring (1) at least one image from one or more source images, each source image being an image of a scene obtained by an optical sensor of an imaging system (2), said method is implemented by a machine, a vehicle or a drone and comprises: - a processing phase P of a reconstructed image M, associated with a source image I, at least some of the pixels of which comprise depth information and/or at least speed information, at the end of the processing phase a restored image C is obtained for each reconstructed image processed, the processing phase comprises an iterative modification of the reconstructed image so as to minimise a function E comprising: • a term D, called a difference term, determined by comparing the reconstructed image convolved by a point spread function (PSF) with the source image, to which the reconstructed image is associated, the PSF describing the response of the imaging system from which the source image used for the comparison is obtained, and • a term A, called anomaly term, representative of defects or anomalies within the reconstructed image, determined from the reconstructed image. 2. Method (1) according to the preceding claim, comprising prior to and/or, respectively, during the processing phase, a step of reconstructing the reconstructed image M or respectively the reconstructed image currently being processed from at least one source image and/or at least one reconstructed image and/or at least one reconstructed image M currently being processed and/or at least one restored image C. 3. Method (1) according to the preceding claim, comprising an extraction of the depth information and/or the speed information of the imaged scene by comparison of at least two source images I or of at least two reconstructed images M or of at least two reconstructed images M currently being processed or of at least two restored images C or of at least one source image with at least one restored image or with a reconstructed image or with a reconstructed image currently being processed. 4. Method (1) according to claim 2 or 3, in which the reconstruction step comprises a convolution of the source image I, with which the reconstructed image is associated, by an inverse function of the PSF, called iPSF, describing the response of the imaging system from which the source image, convolved by the iPSF, is obtained. 5. Method (1) according to any one of claims 2 to 4, in which the reconstruction step comprises a movement of blocks: - according to a speed field, of a source image I, of a reconstructed image M, of a reconstructed image M being processed or of a restored image C, and/or - according to a depth field, of a reconstructed image M, of a reconstructed image M being processed or of a restored image C. 6. Method (1) according to any one of the preceding claims, in which the term D is determined by comparison of: - the reconstructed image M convolved with the PSF or the reconstructed image M being processed convolved with the PSF, with - the source image I, to which the reconstructed image is associated. 7. Method (1) according to the preceding claim, comprising, prior to the comparison of the reconstructed image M convolved with the PSF or of the reconstructed image M currently being processed convolved with the PSF with the source image I, with which the reconstructed image is associated, a spreading, at least locally, according to a speed field and/or according to an acquisition duration of said source image I, of pixel(s) or group(s) or block(s) of pixels of said reconstructed image convolved with the PSF or of said reconstructed image currently being processed convolved with the PSF. 8. Method (1) according to any one of the preceding claims, in which the processing phase P comprises, prior to the convolution of the reconstructed image M or of the reconstructed image M currently being processed with the PSF, a selection of the PSF. 9. Method (1) according to any one of the preceding claims, in which the iterative modification (3) of the reconstructed image M ends when the function E, or a combination of partial derivatives of the function E with respect to the reconstructed image M being processed, is less than a minimization threshold, or when a certain number of iterations of the iterative modification of the reconstructed image is reached; the at least one reconstructed image thus modified is restored. 10. Method (1) according to any one of the preceding claims, in which the reconstructed image M is obtained by merging at least two reconstructed images. 11. Method (1) according to any one of the preceding claims, in which the processing phase P comprises a step of merging at least two restored images C to provide at least one restored image. 12. Method (1) according to any one of the preceding claims, in which, for an imaging system (2) considered, the PSF depends on: - an orientation of an optical axis of the imaging system considered relative to at least one other optical axis of another imaging system, and/or - a relative position of the imaging system considered with respect to at least one other imaging system, and/or - a distance between the sensor of the imaging system considered and an object imaged by the sensor of the imaging system considered, and/or - a distance between a part of the imaging system considered and the at least one sensor, and/or - a distance between a part of the imaging system considered and an object imaged by the sensor of the imaging system considered, and/or - a state of the imaging system, such as a zoom or focus or numerical aperture setting of the imaging system, and/or - of the pixel or of a group or of a block of pixels of a source image or of a reconstructed image or of a restored image and/or of a photosite of the sensor of the imaging system considered, and/or - of one or more pieces of information contained in a pixel or a group or block of pixels of an image, and/or - one or more angles between the optical sensor of the imaging system considered and the imaging system. 13. Method (1) according to claim 2 or any one of claims 3 to 12 taken in combination with claim 2, in which the at least one source image I comprises a set of source images consisting of a temporal succession of source images (Itl, It2...) acquired at different times (tl, t2...), each processing phase (Pl, P2...), each reconstruction step, each reconstructed image (Ml, M2...) and each restored image (Cl, C2...) are associated with a source image obtained at a given time t. 14. Method (1) according to claim 13, in which the reconstructed image Mt, associated with the source image It, is obtained by comparison: - of an image obtained by displacements of blocks of the restored image Ct-1, or of the reconstructed image Mt-1 or of the reconstructed image Mt-1 currently being processed, at the processing phase Pt-1, with - the source image It, to which the reconstructed image Mt is associated, convolved by the inverse function of the PSF, called iPSF, describing the response of the imaging system (2) from which the source image It is obtained. 15. Method (1) according to claim 13 or 14, in which the reconstructed image Mt is obtained from the source image It with which the reconstructed image Mt is associated. 16. Method (1) according to any one of claims 13 to 15, in which the reconstructed image Mt, associated with the source image It, is obtained from the restored image Ct-1 obtained in the processing phase Pt-1. 17. Method (1) according to claim 2 or any one of claims 3 to 12 taken in combination with claim 2, in which the at least one source image I comprises a set of source images (Id, Ig...) acquired by different imaging systems (d, g...), each processing phase (Pd, Pg...), each reconstructed image (Md, Mg...) and each restored image (Cd, Cg...) are associated with a source image I acquired by a different imaging system (2); the imaging systems being arranged so as to acquire, at least in part, the same scene. 18. Method (1) according to the preceding claim, in which the step of reconstructing the reconstructed image Mt is carried out from: - of a source image Id or a rendered image Cd, and - from a source image Ig or a rendered image Cg. 19. Method (1) according to claim 17 or 18 taken in combination with any one of claims 13 to 16, in which each of the source images of the set of source images (Idtl, Idt2, Igtl, Igt2...) is acquired by a separate imaging system (2) (d, g...) at a given time (tl, t2...), each processing phase (Pdtl, Pdt2, Pgtl, Pgt2...), each reconstructed image (Mdtl, Mdt2, Mgtl, Mgt2...) and each restored image (Cdtl, Cdt2, Cgtl, Cgt2...) are associated with a source image (Idtl, Idt2, Igtl, Igt2...) obtained by a given imaging system and at a given time t. 20. Data processing device comprising means arranged and/or programmed and/or configured to implement the method (1) according to any one of claims 1 to 19. 21. Computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method (1) according to any one of claims 1 to 19. 22. A computer-readable medium comprising instructions which, when executed by a computer, cause the latter to implement the method (1) according to any one of claims 1 to 19.