FR3160493A1 - Method and device for learning a depth prediction model associated with a stereoscopic vision system and insensitive to occlusion. - Google Patents

Abstract

A method or device for learning a depth prediction model implemented by a neural network associated with a vision system. The method comprises generating (32) depth maps associated with images acquired by each camera of the vision system and generating (33) other images and other depth maps from the received images and the depth maps by changing the reference frame. A visibility mask is determined (34) from the generated images, and a zero value is assigned (35) to the pixels of the third and fourth images not included in the visibility mask. The depth prediction model is then learned (36) by minimizing a loss error determined by comparing the depth maps to the other depth maps and by comparing the received images to the generated images. Abstract: Figure 3

Description

Method and device for learning a depth prediction model associated with a stereoscopic vision system and insensitive to occlusion.

The present invention relates to methods and devices for determining a depth by a vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring a distance separating an object from a vehicle carrying a vision system.

Technological background

Many modern vehicles are equipped with so-called ADAS (Advanced Driver Assistance System). ADAS are passive and active safety systems designed to eliminate human error in the operation of all types of vehicles. ADAS uses advanced technologies to assist the driver while driving and thus improve their performance. ADAS uses a combination of sensor technologies to perceive the environment around a vehicle, then provides information to the driver or influences certain vehicle systems.

There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control, and automatic parking systems.

ADAS systems embedded in a vehicle are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras are used to detect and locate other road users or possible obstacles around a vehicle in order to, for example:
• to adapt the vehicle's lighting according to the presence of other users;
• to automatically regulate the vehicle speed;
• to act on the braking system in the event of a risk of impact with an object.

Predicting the distance between an object and a vehicle, i.e. the depth associated with a pixel in an image acquired by a camera, the pixel representing the object, requires high precision in order to feed ADAS. Predicting the depth associated with a pixel notably calls upon a depth prediction model, the depth prediction model being adapted to the vehicle's onboard vision system. Such a prediction model is commonly implemented by a neural network, which is notably learned in a training phase in order to predict accurate depths in relation to the onboard vision system and the type of environment in which the vehicle operates. A training phase then requires data, for example annotated data from libraries or annotated via other more precise onboard systems such as a LIDAR®. However, this type of learning requires a massive accumulation of data or the presence of this other onboard system which the vehicle does not always have. It is then possible to carry out a learning phase of the depth prediction model from data acquired by the on-board vision system, but existing solutions are not suitable for the configuration of any on-board vision system in a vehicle.

Furthermore, objects present in a three-dimensional scene observed by a stereoscopic vision system, i.e. a vision system comprising at least two cameras, may not be visible on certain images acquired by the stereoscopic vision system, these objects then being occluded from the point of view of a camera of the stereoscopic vision system. The presence of occluded objects is then likely to generate errors and distort the learning of a depth prediction model.

Summary of the present invention

An object of the present invention is to solve at least one of the problems of the technological background described above.

Another object of the present invention is to improve the quality of the data resulting from the processing of an image acquired by a vision system, in particular by a depth prediction model implemented by a neural network associated with a stereoscopic vision system.

Another object of the present invention is to improve road safety, in particular by improving the operational safety of ADAS systems supplied by data obtained from a camera, in particular a wide-angle camera.

According to a first aspect, the present invention relates to a method for learning a depth prediction model implemented by a convolutional neural network associated with a stereoscopic vision system on board a vehicle, the stereoscopic vision system comprising a first camera and a second camera arranged so as to each acquire an image of a three-dimensional scene from a different point of view,
said method being implemented by at least one processor, and being characterized in that it comprises the following steps:
- reception of a first image and a second image acquired by the first camera and the second camera respectively at the same acquisition time instant;
- generating a first depth map comprising depths associated with a set of pixels of the first image and a second depth map comprising depths associated with a set of pixels of the second image, the depths being predicted with the depth prediction model from the first and second images;
- generating a third image and a third depth map from the first image, the first depth map and extrinsic parameters of the stereoscopic vision system, the third depth map comprising depths associated with a set of pixels of the third image, and generating a fourth image and a fourth depth map from the second image, the second depth map and extrinsic parameters of the stereoscopic vision system, the fourth depth map comprising depths associated with a set of pixels of the fourth image;
- determining a visibility mask as the union of a first visibility mask and a second visibility mask, each pixel of the third image resulting from at least one pixel of the first image belonging to the first visibility mask and each pixel of the fourth image resulting from at least one pixel of the second image belonging to the second visibility mask;
- assigning a zero value to the pixels of the third and fourth images not included in the visibility mask;
- learning the depth prediction model by minimizing a loss error, the loss error being determined from:
• a first consistency error determined by comparing the first and fourth depth maps,
• a second consistency error determined by comparing the second and third depth maps, and
• a photometric error determined by comparing the first and fourth images and by comparing the second and third images.

According to a variant, the first consistency error and the second consistency error are determined respectively by the following function:

With :
• corresponding to the first consistency error , respectively the second consistency error ,
• a depth of one pixel obtained from the first depth map, respectively from the second depth map, for a pixel , And
• a depth of one pixel obtained from the fourth depth map, respectively from the third depth map, for the pixel .

According to another variant, the photometric error is determined by the following function:

With :
• the first photometric error noted , respectively the second photometric error noted , being a pixel defined by its coordinates in an image,
• a pixel value in the first image, respectively second image,
• a pixel value in the fourth image, respectively third image,
• SSIM a function that takes into account a local structure, and
• a weighting factor depending in particular on the type of environment.

According to a further variant, the loss error is further determined from a construction error determined by the following function:

With :
• the construction error for a pixel p of the third image, respectively the construction error for a pixel p of the fourth image,
• is a depth of one pixel obtained from the third depth map, respectively obtained from the fourth depth map;
• is a parameter matrix;
• is the order of a smoothing gradient;
• an L1 norm of second-order depth gradients is calculated with =1, and =2 ;
• And are the dimensions of the images;
• is a hyperparameter dependent on the environment in which the vehicle operates; and
• is a pixel value in the third image, respectively fourth image.

According to a further variant, the loss error is determined by the following function:

With :
• the loss error,
• the first photometric error for a pixel , being a pixel defined by its coordinates in an image,
• the second photometric error for the pixel ,
• the first consistency error for the pixel ,
• the second consistency error for the pixel ,
• the construction error for a pixel of the third image, and
• the construction error for a pixel of the fourth image.

Alternatively, the third and fourth depth maps are generated by the following function:

With :
• the coordinates of a pixel of the third depth map, respectively of the fourth depth map,
• a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector,
• an intrinsic matrix of the first camera, respectively to the second camera,
• an intrinsic matrix of the second camera, respectively to the first camera,
• an extrinsic matrix of the stereoscopic vision system, and
• a projection function into the three-dimensional scene of a pixel depending on its coordinates in the first image, respectively second image, and the depth which is associated with it in the first depth map, respectively second depth map.

According to yet another variant, optical axes of the first and second cameras are not coplanar.

According to a second aspect, the present invention relates to a device configured to learn a depth prediction model by a vision system on board a vehicle, the device comprising a memory associated with at least one processor configured to implement the steps of the method according to the first aspect of the present invention.

According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

Such a computer program may use any programming language and be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording medium or a hard disk.

Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from a network such as the Internet.

Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question.

Brief description of the figures

Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 4, in which:

FIG. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

FIG. 2 illustrates a flowchart of the different steps of a method for determining the depth of a pixel in an image using a depth prediction model associated with a vision system embedded in the vehicle of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention;

FIG. 3 illustrates a flowchart of the different steps of a process for learning the depth prediction model used in the process of the FIG. 2 , according to a particular and non-limiting embodiment of the present invention; and

FIG. 4 schematically illustrates a device configured to learn a depth prediction model by an on-board vision system in the vehicle of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention.

Description of examples of implementation

A method and a device for learning a depth prediction model implemented by a convolutional neural network associated with a stereoscopic vision system on board a vehicle will now be described in what follows with joint reference to figures 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.

The terms “first(s)”, “second(s)” (or “first(s)”, “second(s)”), etc. are used in this document by arbitrary convention to identify and distinguish different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

For the entire description, reception of an image means the reception of data representative of an image. Similarly, generation of an image means the generation of data representative of an image and generation of a depth map means the generation of data representative of a depth map. These shortcuts are only intended to simplify the description; however, since the processes are implemented by one or more processors, it is obvious that the input and output data of the different stages of a process are computer data.

According to a particular and non-limiting example of embodiment of the present invention, a depth prediction model associated with a stereoscopic vision system comprising two cameras is learned in a learning phase.

Indeed, the method relating to the learning phase includes the generation of depth maps associated with images acquired by each camera of the vision system and the generation of other images and other depth maps from the received images and the depth maps by changing the reference frame.

A visibility mask is then determined from the generated images and a zero value is assigned to the pixels in the third and fourth images not included in the visibility mask.

The depth prediction model is then learned by minimizing a loss error determined by comparing the depth maps to other depth maps and by comparing the received images to the generated images.

There FIG. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, an electric motor(s) or a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

The vehicle 10 advantageously comprises at least two on-board cameras, a first camera 11 and a second camera 12, configured to acquire images of a three-dimensional scene taking place in the environment of the vehicle 10 from distinct observation positions. The first camera 11 and the second camera 12 form a stereoscopic vision system when used together as illustrated in the FIG. 1 . The first camera 11 forms a monoscopic vision system when used alone, likewise the second camera 12 forms another monoscopic vision system when used alone. The present invention however extends to any vision system comprising at least two cameras, for example 2, 3 or 5 cameras.

The intrinsic parameters of the first camera 11 characterize the transformation which associates, for an image point, subsequently called a “point”, its three-dimensional coordinates in the reference frame of the first camera 11 with the pixel coordinates in an image acquired by the first camera 11. These parameters do not change if the first camera 11 is moved. The intrinsic parameters of the first camera 11 include in particular a first focal length f1 associated with the first camera 11.

The intrinsic parameters of the second camera 12 characterize, for their part, the transformation which associates, for an image point, its three-dimensional coordinates in the reference frame of the second camera 12 with the pixel coordinates in an image acquired by the second camera 12. These parameters do not change if the second camera 12 is moved. The intrinsic parameters of the second camera 12 include in particular a second focal length f2 associated with the second camera 12.

Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account, that their correction is processed at the time of image acquisition or at the time of calibration.

These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.

A first marker is associated with the first camera 11:
- the direction of the x axis is defined horizontal and normal to the optical axis C1 of the first camera 11. The distance B separating the optical center of the first camera 11 from the projection of the optical center of the second camera 12 on the horizontal plane passing through the optical center of the first camera 11 is called the reference base (in English “baseline”);
- the direction of the y axis is defined vertical and normal to the optical axis C1 of the first camera 11;
- the direction of the z axis is defined orthogonal to the directions of the x and y axes.
The three axes x, y and z thus form an orthonormal reference frame.

The optical axis C1 of the first camera 11 and the optical axis C2 of the second camera 12 are not necessarily parallel or even included in the same plane.

According to a variant, optical axes of the first and second cameras are not coplanar.

The extrinsic parameters related to the position of cameras 11, 12 are the following parameters:
- three translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and
- three rotations in the x, y and z directions: θx, θy and θz.

An extrinsic matrix of the vision system then includes the previously defined extrinsic parameters.

The extrinsic parameters are determined, for example, during a calibration phase of the stereoscopic vision system comprising the first camera 11 and the second camera 12.

A major constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measuring range of 200 meters, the reference base must reach 60 cm for the cameras commonly used in this field.

The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic view of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic view of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic view of the scene by the stereoscopic vision system composed of the two cameras 11, 12.

An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system comprising the second camera 12.

Field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system comprising the first camera 11.

Finally, field 19 is not visible to any of the cameras. The part of the scene present in this field 19 is therefore not observable.

According to a particular exemplary embodiment, the field of vision of the second camera 12 covers at least half of the field of vision of the first camera 11.

It is obvious that it is possible to use such a stereoscopic vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with differently placed and oriented cameras.

The images acquired by the cameras 11, 12 at an acquisition time instant are presented in the form of data representing pixels characterized by:
- coordinates in each image; and
- data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric coordinates (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

Each pixel in the acquired image represents an object in the three-dimensional scene present in the camera's field of vision. In fact, a pixel in the acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are emitted or reflected, captured by a photosensitive sensor in the camera after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image faithfully representing the physical object observed by the camera. An image point previously presented is thus a point on a surface of an object in the three-dimensional scene.

The images acquired by cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are for example:
- buildings;
- road infrastructure;
- other stationary users, for example a parked vehicle; and/or
- other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

According to a particular exemplary embodiment, a field of view of the first camera 11 covers at least half of a field of view of the second camera 12 and a field of view of the second camera 12 covers at least half of a field of view of the first camera 11. In other words, more than half of the pixels of an image acquired by the first camera 11 correspond to an object of the three-dimensional scene seen by the second camera 12, pixels of an image acquired by the second camera 12 also corresponding to this object of the three-dimensional scene. Similarly, more than half of the pixels of an image acquired by the second camera 12 correspond to an object of the three-dimensional scene seen by the first camera 11, pixels of an image acquired by the first camera 11 also corresponding to this object of the three-dimensional scene.

The images acquired by the first camera 11 and by the second camera 12 are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

A method for determining a depth by a vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, that is to say by a processor, a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the vision system of the vehicle 10.

There FIG. 2 illustrates a flowchart of the different steps of a method 2 for determining the depth of a pixel of an image by depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, for example in the vehicle 10 of the FIG. 1 , according to a particular and non-limiting embodiment of the present invention. The method 2 is for example implemented by a device of the vision system on board the vehicle 10 or by the device 4 of the FIG. 4 .

In a step 21, data representative of an image acquired by the first camera 11 and of an image acquired by the second camera 12 are received.

In a step 22, depths associated with a set of pixels of one of the received images are determined by the depth prediction model from the two received images.

Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object of the three-dimensional scene with which a pixel is associated, the determination of a depth of a pixel then corresponding to a measurement of a distance separating an object from the vehicle carrying the vision system.

If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.

There FIG. 3 illustrates a flowchart of the different steps of a method for learning the depth prediction model used in a method for determining the depth of a pixel of an image, for example in method 2 of the FIG. 2 , according to a particular and non-limiting embodiment of the present invention.

The learning method 3 is for example implemented by the device on board the vehicle 10 implementing the method for determining a depth by the vision system on board a vehicle or by the device 4 of the FIG. 4 .

In a step 31, a first image and a second image are received, the first image being acquired by the first camera 11 at an acquisition time instant and the second image being acquired by the second camera at the same acquisition time instant.

According to a particular exemplary embodiment, the first image and second image have the same definition, that is to say they comprise the same number of pixels, have the same number of pixels according to their height and the same number of pixels according to their width.

According to another particular embodiment, the first image and second image do not have the same definition. An additional step then consists of resizing or cropping them to obtain a first image and a second image of the same definition.

In a step 32, a first depth map comprising depths associated with a set of pixels of the first image is determined. The depths associated with the pixels of the first image are predicted with the depth prediction model from the first and second images.

Similarly, a second depth map comprising depths associated with a set of pixels of the second image is determined. The depths associated with the pixels of the second image are also predicted with the depth prediction model from the first and second images.

The depth prediction model, implemented by a convolutional neural network, is known to those skilled in the art and is for example presented in the document “Unifying Flow, Stereo and Depth Estimation” written by Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao and Andreas Geiger, published in July 2023, suitable for any stereoscopic vision system including those comprising cameras whose optical axes are not included in the same plane.

Thus, a predicted depth is associated with each pixel of a set of pixels of the first image, the first depth map comprising the coordinates of each pixel of the set of pixels of the first image and a depth associated with each pixel recorded for example in a correspondence table, for example in a memory accessible to the processor implementing this learning method 3. This correspondence table then contains pairs (coordinates of a pixel of image 1; predicted depth for this pixel).

Similarly, a predicted depth is associated with each pixel of a set of pixels of the second image, the second depth map comprising the coordinates of each pixel of the set of pixels of the second image and a depth associated with each pixel recorded for example in a correspondence table, for example in a memory accessible to the processor implementing this learning method 3. This correspondence table then contains pairs (coordinates of a pixel of image 2; predicted depth for this pixel).

In a step 33, a third image and a third depth map are generated from the first image, the first depth map and extrinsic parameters of the stereoscopic vision system, the third depth map comprising depths associated with a set of pixels of the third image.

The generation of the third depth map from the first image, the first depth map and extrinsic parameters of the stereoscopic vision system consists of:
• determining spatial coordinates in the three-dimensional scene, in a reference system associated with the first camera 11, of a point associated with a first pixel of the first depth map from the coordinates of the first pixel in the first image and the depth which is associated with this first pixel, its coordinates and depth being recorded in the first depth map. It should be noted that the coordinates of the first pixel in the first image comprise two components while the spatial coordinates of the point associated with the first pixel comprise three components;
• the determination of spatial coordinates in the three-dimensional scene of the point associated with the first pixel in a reference frame associated with the second camera 12, in other words a change of reference frame from that associated with the first camera 11 to that associated with the second camera 12, using the extrinsic parameters of the stereoscopic vision system; and
• the projection of the point associated with the pixel in the image plane of the second camera 12, this image plane corresponding to that of the second image, making it possible to determine the arrival coordinates of the first pixel of the first image in an image such as the second camera 12 would have acquired it. The image plane of a camera corresponds to a plane defined in the frame of reference of the camera, normal to the optical axis of the camera and located at the first focal length of the camera.
The third depth map then includes the arrival coordinates of the first pixels and the depth associated with the first pixel.

The arrival coordinates of a pixel in the third depth map are thus determined and are the same as the arrival coordinates of a pixel in the third image. In the third depth map, the depth of a pixel corresponds to a depth determined from pixel depths in the first depth map, while in the third image, the value of a pixel corresponds to a value determined from pixel values in the first image.

Similarly, a fourth image and a fourth depth map are generated from the second image, the second depth map, and extrinsic parameters of the stereoscopic vision system, the fourth depth map comprising depths associated with a set of pixels of the fourth image.

According to a particular exemplary embodiment, the third depth map is generated by the following function:

With :
• the coordinates of a pixel of the third depth map,
• a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector,
• the intrinsic matrix of the first camera 11,
• the intrinsic matrix of the second camera 12,
• an extrinsic matrix of the stereoscopic vision system, and
• a projection function into the three-dimensional scene of a first pixel depending on its coordinates in the first image and the depth which is associated with it in the first depth map.
This function thus determines the arrival coordinates of a pixel of an image acquired by the first camera 11 in an image as it would be acquired by the second camera 12.

The determination of the fourth depth map is similar to the determination of the third depth map, in particular the cameras are reversed. The arrival coordinates of a pixel in the fourth depth map are thus determined and are the same as the arrival coordinates of a pixel in the fourth image. In the fourth depth map, the depth of a pixel corresponds to a depth determined from pixel depths in the second depth map, while in the fourth image, the value of a pixel corresponds to a value of a pixel determined from pixel values in the second image.

Thus, the previous function for determining the arrival coordinates of a first pixel of the first depth map is applicable for determining the arrival coordinates of a second pixel of the second depth map, the function being adapted as follows:

With :
• the coordinates of a pixel of the fourth depth map,
• a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector,
• the intrinsic matrix of the second camera 12,
• the intrinsic matrix of the first camera 11,
• an extrinsic matrix of the stereoscopic vision system, and
• a projection function into the three-dimensional scene of a second pixel depending on its coordinates in the second image and the depth which is associated with it in the second depth map.
This function thus determines the arrival coordinates of a pixel of an image acquired by the second camera 12 in an image as it would be acquired by the first camera 11.

According to an alternative embodiment, values associated with pixels of the third and fourth images, that is to say colorimetric values of the pixels of the third and fourth images, are obtained by interpolation of the values of the pixels of the first image and respectively of the second image using a function such as torch.nn.functional.grid_sample() in Python® language which requires as arguments the arrival coordinates of the pixels of the third and fourth images and the values of the preceding pixels in the first image, respectively second image.

In a step 34, a visibility mask is determined. This visibility mask is like the union of a first visibility mask and a second visibility mask, each pixel of the third image resulting from at least one pixel of the first image belonging to the first visibility mask and each pixel of the fourth image resulting from at least one pixel of the second image belonging to the second visibility mask. In other words, the first visibility mask includes the set of pixels of the third image which have an antecedent in the first image during the step of generating the third image, that is to say the set of pixels of the third image whose coordinates are arrival coordinates of a pixel of the first image. Similarly, the second visibility mask includes the set of pixels of the fourth image which have an antecedent in the second image during the step of generating the fourth image, that is to say the set of pixels of the fourth image whose coordinates are arrival coordinates of a pixel of the second image.

Conversely, the pixels not included in the first visibility mask are pixels which have no antecedent in the first image acquired by the first camera 11, they then correspond to objects occluded from the point of view of the second camera 12 and the pixels not included in the second visibility mask are pixels which have no antecedent in the second image acquired by the second camera 12, they then correspond to objects occluded from the point of view of the first camera 11.

The determination of the visibility mask is then the union of the first and second visibility masks, that is to say that coordinates corresponding to those of a pixel of the third image which has no antecedent in the first image and also corresponding to those of a pixel of the fourth image which has no antecedent in the second image define coordinates of a pixel not included in the visibility mask. Conversely, if coordinates correspond to those of a pixel of the third image which has a antecedent in the first image and/or to those of a pixel of the fourth image which has a antecedent in the second image then these coordinates define coordinates of a pixel included in the visibility mask.

The visibility mask is determined, for example, by using the torch.nn.functional.grid_sample() function in Python®, which requires as arguments the pixel coordinates of the third and fourth depth maps and the depths determined in the second and first depth maps.

In a step 35, a zero value is assigned to the pixels of the third and fourth images not included in the visibility mask. These pixels of the third and fourth images have no antecedent in the first image, respectively in the second image. A pixel with a zero value corresponds, for example, to a black pixel. In the case of several channels, each channel associated with the pixel is assigned a zero value.

In a step 36, the depth prediction model is learned by minimizing a loss error, the loss error being determined from:
• a first consistency error determined by comparing the first and fourth depth maps,
• a second consistency error determined by comparing the second and third depth maps, and
• a photometric error determined by comparing the first and fourth images and by comparing the second and third images.

According to a particular exemplary embodiment, the first consistency error and the second consistency error are determined respectively by the following function:

With :
• corresponding to the first consistency error , respectively the second consistency error , for a pixel ,
• a depth of one pixel obtained from the first depth map, respectively from the second depth map, for a pixel , And
• a depth of one pixel obtained from the fourth depth map, respectively from the third depth map, for the pixel .

Other functions can be used to perform this comparison, for example by taking the absolute value of the difference in depths. Indeed, it is important that the result of the depth comparison is, for each pixel, a positive value.

According to a first particular exemplary embodiment, the photometric error is determined from a first photometric error by comparing pixel values of the first and fourth images and from a second photometric error determined by comparing pixel values of the second and third images, the loss error being determined from the first and second photometric errors.

A photometric error is, for example, presented in the paper "Digging Into Self-Supervised Monocular Depth Estimation" by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow published in August 2019 and is determined by the following function:

With :
• the first photometric error noted , respectively the second photometric error noted , being a pixel defined by its coordinates in an image,
• a pixel value in the first image, respectively second image,
• a pixel value in the fourth image, respectively third image,
• SSIM a function that takes into account a local structure, and
• a weighting factor depending in particular on the type of environment.

According to this particular embodiment, the loss error is determined by the following function:

With :
• the loss error,
• the first photometric error for a pixel , being a pixel defined by its coordinates in an image,
• the second photometric error for the pixel ,
• the first consistency error for the pixel , And
• the second consistency error for the pixel .

According to a second particular exemplary embodiment, the loss error further comprises a construction error, each of the components of which is determined by the following function:

With :
• the component for a pixel p of the third image, respectively the component for a pixel p of the fourth image,
• is a depth of one pixel obtained from the third depth map, respectively obtained from the fourth depth map;
• is a parameter matrix;
• is the order of a smoothing gradient;
• an L1 norm of second-order depth gradients is calculated with =1, and =2 ;
• And are the dimensions of the images;
• is a hyperparameter dependent on the environment in which the vehicle operates; and
• is a pixel value in the third image, respectively fourth image.

This function is generally used to deal with discontinuity at the edge of objects (in English "edge aware smoothness").

The loss error includes, for example, the consistency errors, photometric errors and the construction error determined above:

With :
• the loss error,
• the first photometric error for a pixel , being a pixel defined by its coordinates in an image,
• the second photometric error for the pixel ,
• the first consistency error for the pixel ,
• the second consistency error for the pixel ,
• the component in the third image for the pixel ,
• the component in the fourth image for the pixel .

It should be noted that the pixels compared are those with similar or equal coordinates in both the acquired and generated images.

Training the depth prediction model consists of adjusting input parameters of the convolutional neural network to minimize the previously calculated loss error.

Furthermore, an occluded or non-visible object in the field of view of one camera and masked in the field of view of the other camera does not impact the loss error, the use of a visibility mask making this learning process insensitive to occlusions. Thus, the depth prediction model used for the depth prediction of a pixel of an image acquired by one of the cameras of the stereoscopic vision system is made reliable thanks to this learning process.

This learning is carried out from data acquired by the on-board vision system and therefore does not require data annotated by another on-board system or storage of a library of training images. In addition, the training data is representative of the data received when the system is in operation or in production, in fact the training data is representative of real environments in which the vehicle carrying the stereoscopic vision system evolves or moves, this training data is therefore particularly relevant.

ThereFIG. 4schematically illustrates a device 4 configured to learn a depth prediction model by a vision system embedded in a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device embedded in the first vehicle 10, for example a computer associated with the stereoscopic vision system.

The device 4 is for example configured for the implementation of the operations described with regard to figures 1 and 4 and/or steps described with regard to figures 2 and 3. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and/or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and/or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input/output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and/or non-volatile memory and/or comprises a memory storage device which may comprise volatile and/or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

The computer code of the embedded software(s) including the instructions to be loaded and executed by the processor is for example stored in memory 41.

According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and/or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input/output ports.

According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces:
- RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced;
- USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French);
HDMI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French);
- LIN interface (from the English “Local Interconnect Network”).

According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and/or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO/IEC 802-3 standard).

According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and/or other peripherals 460 via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for determining the depth of a pixel of an image acquired by a vision system, and/or for measuring a distance separating an object from a vehicle carrying a vision system, the depth and/or the distance being predicted and/or measured via a depth prediction model learned according to the learning method described above, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of the FIG. 4 .

Claims (10)

Method for learning a depth prediction model implemented by a convolutional neural network associated with a stereoscopic vision system on board a vehicle (10), the stereoscopic vision system comprising a first camera (11) and a second camera (12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view,
said method being implemented by at least one processor, and being characterized in that it comprises the following steps:
- reception (31) of a first image and a second image acquired by the first camera (11) and the second camera (12) respectively at the same acquisition time instant;
- generating (32) a first depth map comprising depths associated with a set of pixels of the first image and a second depth map comprising depths associated with a set of pixels of the second image, the depths being predicted with the depth prediction model from the first and second images;
- generation (33) of a third image and a third depth map from the first image, the first depth map and extrinsic parameters of the stereoscopic vision system, the third depth map comprising depths associated with a set of pixels of the third image, and generation of a fourth image and a fourth depth map from the second image, the second depth map and extrinsic parameters of the stereoscopic vision system, the fourth depth map comprising depths associated with a set of pixels of the fourth image;
- determining (34) a visibility mask as the union of a first visibility mask and a second visibility mask, each pixel of the third image resulting from at least one pixel of the first image belonging to the first visibility mask and each pixel of the fourth image resulting from at least one pixel of the second image belonging to the second visibility mask;
- assignment (35) of a zero value to the pixels of the third and fourth images not included in the visibility mask;
- learning (36) the depth prediction model by minimizing a loss error, the loss error being determined from:
• a first consistency error determined by comparing the first and fourth depth maps,
• a second consistency error determined by comparing the second and third depth maps, and
• a photometric error determined by comparing the first and fourth images and by comparing the second and third images. Method according to claim 1, wherein the first consistency error and the second consistency error are determined respectively by the following function:

With :
• corresponding to the first consistency error , respectively the second consistency error ,
• a depth of one pixel obtained from the first depth map, respectively from the second depth map, for a pixel , And
• a depth of one pixel obtained from the fourth depth map, respectively from the third depth map, for the pixel . Method according to claim 1 or 2, for which the photometric error is determined by the following function:

With :
• the first photometric error noted , respectively the second photometric error noted , being a pixel defined by its coordinates in an image,
• a pixel value in the first image, respectively second image,
• a pixel value in the fourth image, respectively third image,
• SSIM a function that takes into account a local structure, and
• a weighting factor depending in particular on the type of environment. Method according to one of claims 1 to 3, for which the loss error is further determined from a construction error determined by the following function:

With :
• the construction error for a pixel p of the third image, respectively the construction error for a pixel p of the fourth image,
• is a depth of one pixel obtained from the third depth map, respectively obtained from the fourth depth map;
• is a parameter matrix;
• is the order of a smoothing gradient;
• an L1 norm of second-order depth gradients is calculated with =1, and =2 ;
• And are the dimensions of the images;
• is a hyperparameter dependent on the environment in which the vehicle operates; and
• is a pixel value in the third image, respectively fourth image. The method of claim 4, wherein the loss error is determined by the following function:

With :
• the loss error,
• the first photometric error for a pixel , being a pixel defined by its coordinates in an image,
• the second photometric error for the pixel ,
• the first consistency error for the pixel ,
• the second consistency error for the pixel ,
• the construction error for a pixel of the third image, and
• the construction error for a pixel of the fourth image. Method according to one of claims 1 to 5, for which the third and fourth depth maps are generated by the following function:

With :
• the coordinates of a pixel of the third depth map, respectively of the fourth depth map,
• a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector,
• an intrinsic matrix of the first camera (11), respectively to the second camera (12),
• an intrinsic matrix of the second camera (12), respectively to the first camera (11),
• an extrinsic matrix of the stereoscopic vision system, and
• a projection function into the three-dimensional scene of a pixel depending on its coordinates in the first image, respectively second image, and the depth which is associated with it in the first depth map, respectively second depth map. Method according to one of claims 1 to 6, for which optical axes of the first and second cameras are not coplanar. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor. Device (4) configured to learn a depth prediction model by a vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured to implement the steps of the method according to any one of claims 1 to 7. Vehicle (10) comprising the device (4) according to claim 9.