FR3127165A1 - Method and device for aiding driving in augmented reality - Google Patents

Abstract

The present invention relates to an augmented reality driving aid device and method for a first vehicle traveling in a road environment, said method comprising the following steps: reception of at least one piece of information representative of said road environment captured by a sensor module (10) on board said first vehicle; data processing of said at least one piece of road environment information for recognition in said road environment of at least one piece of identification information relating to a second vehicle traveling in front of said first vehicle; determination of an image representative of the second vehicle as a function of said at least one piece of identification information; and displaying said representative image of said second vehicle on a screen (201) of said first vehicle. Figure for abstract: Figure 2

Description

Method and device for aiding driving in augmented reality

The present invention relates to the field of intelligent vehicles, and in particular the field of autonomous vehicles.

The present invention relates more particularly to the field of driver information systems, also known as infotainment.

The object of the present invention here relates to a method and a device for assisting the driving in augmented reality of a first vehicle traveling in a road environment behind a second vehicle.

Technology background

Vehicles conventionally comprise a dashboard having one or more display screens providing the driver with a plurality of so-called intrinsic information of the vehicle.

This intrinsic information relates to the vehicle itself and includes, for example, information on the operation of the engine and/or the driving parameters of the vehicle; these concern, for example, the instantaneous speed of the vehicle, the rotational speed of the main engine shaft, the mileage, the fuel level, the oil level or the temperature of the coolant, etc.

This intrinsic information also includes many other safety-related information such as, for example, alarm, warning and signaling indicators.

Contemporary vehicles now include some of them with other screens, for example with a head-up display, displaying other extrinsic information relating, for example, to the road environment of said vehicle.

The display of this extrinsic information very often contributes to the safety of the vehicle and allows the driver to understand the road environment in which it is traveling by improving his perception of the road environment.

Thus, we already know certain ADAS systems (from the English "Advanced Driver-Assistance System" or in French "Advanced Driving Assistance System") which are equipped with an on-board camera and which are configured to display on a screen, for example a head-up screen or HUD (from English “Head Up Display”), an element of the road environment, such as for example a road sign of the road environment and/or an icon or a pictogram representative of a vehicle traveling in front of said vehicle.

The document EP3543059 thus proposes a method for calculating an augmented reality overlay of a vehicle to facilitate the longitudinal guidance of the observing vehicle. In it, an animation graphic is calculated when adjusting the regulation activity of the longitudinal guidance system for the observer vehicle when approaching the vehicle in front.

The document FR3064372 proposes an augmented reality driving assistance device for a vehicle. In this document, provision is made to calculate an approximation of the trajectory of the vehicle, for example as a function of the speed and/or of the steering angle of the latter, and to generate a display signal as a function of this trajectory. A head-up display is then provided providing, in the driver's field of vision, a virtual image representative of the trajectory of said vehicle.

However, the vehicle displays offered in these documents remain unrealistic and consist of the embedding in the screen of a simple icon or a simple pictogram. Very often, this icon or this pictogram is a representation of the observing vehicle and not of the vehicle traveling in front of said observing vehicle.

This makes the current display unrealistic.

For these reasons, the information displayed is not intuitive for the driver.

The Applicant therefore submits that to date there is no augmented reality technology making it possible to enrich the display in a vehicle in order to provide the driver with a simple, realistic and intuitive representation of the information displayed by limiting the necessary cognitive efforts. the driver to perceive and understand this information and his environment.

Summary of the present invention

The present invention aims to improve the situation described above.

An object of the present invention is to solve at least one of the problems of the technological background described above.

One of the objectives of the present invention is in particular to remedy the various drawbacks mentioned above by proposing an augmented reality driving assistance solution for a first vehicle traveling in a road environment which proposes to improve the immersion of the driver.

According to a first aspect, the present invention relates to an augmented reality driving aid method for a first vehicle traveling in a road environment.

According to the invention, the method comprises the following steps:
- reception of at least one piece of information representative of the road environment captured by a sensor module on board the first vehicle;
- data processing of the at least one piece of road environment information for recognition in the road environment of at least one piece of identification information relating to a second vehicle traveling in front of the first vehicle;
- determination of an image representative of the second vehicle as a function of the at least one piece of identification information; And
- display of the representative image of the second vehicle on a screen of the first vehicle.

In an advantageous embodiment of the present invention, the determination of the representative image of the second vehicle is carried out by comparing the at least one piece of identification information with a database comprising for each vehicle of a list of vehicles determined a predetermined image representative of the vehicle.

According to a variant, the at least one piece of identification information relating to a second vehicle comprises a silhouette and/or a model of the second vehicle.

Optionally, the method according to the present invention comprises an update of the database.

Advantageously, the database is in the form of a digital library comprising for each silhouette and/or each vehicle model determined at least one image representative of the vehicle.

Preferably, the image of the second vehicle is representative of a rear face of the second vehicle.

Preferably, the display of the image is of the head-up display type.

According to a second aspect, the present invention relates to an augmented reality driving aid device for a first vehicle traveling in a road environment, the device comprising a memory associated with a processor configured for the implementation of the steps of the method according to the first aspect of the present invention.

According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for the execution of the steps of the method according to the first aspect of the present invention, this in particular when the computer program is executed by at least one processor.

Such a computer program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for the execution of the steps of the method according to the first aspect of the present invention.

On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a ROM memory of the microelectronic circuit type, or even a magnetic recording means or a hard disk.

On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other ways. The computer program according to the present invention can in particular be downloaded from an Internet-type network.

Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the method in question.

Brief description of figures

Other characteristics and advantages of the present invention will emerge from the description of the specific and non-limiting examples of embodiments of the present invention below, with reference to the appended figures 1 to 3, in which:

schematically illustrates a road portion of a road environment on which a first and a second vehicle are traveling according to a particular embodiment of the present invention, the first vehicle traveling behind the second vehicle;

schematically illustrates a driving aid device for a first vehicle, according to a particular embodiment of the present invention;

illustrates a flowchart of the different steps of an augmented reality driving aid method for a first vehicle traveling behind a second vehicle in accordance with the and comprising a driving aid device in accordance with the .

Description of the examples of realization

A method and a device for aiding the driving of a first vehicle 210 will now be described in the following with reference jointly to FIGS. 1 to 3. The same elements are identified with the same reference signs throughout the description that follows.

There illustrates a road environment 1 comprising a portion of road on which a first vehicle 200 and a second vehicle 300 travel.

In this example, the first vehicle 200 travels behind a second vehicle 300.

The road portion of this road environment 1 is for example located in an urban environment, on a departmental or national road, on a road with two traffic lanes or on a motorway.

The first vehicle 200 corresponds for example to a vehicle with a heat engine, with electric motor(s) or even a hybrid vehicle with a heat engine and one or more electric motors. The vehicle 200 thus corresponds for example to a land vehicle, for example an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 200 corresponds to an autonomous vehicle or not, that is to say a vehicle circulating according to a determined level of autonomy or under the total supervision of the driver.

It is known as explained in the preamble that the prior art already provides a display of a simple pictogram representing the second vehicle 300 traveling in front of the first vehicle 200. On the other hand, this pictogram in the prior art is unique and does not correspond to a realistic representation of the second vehicle 300, which is misleading and does not allow the driver to have on his screen 201 realistic and intuitive information.

It is therefore appropriate here to improve the realism of the information displayed on one of the screens 201 of the dashboard of the first vehicle 200 and to provide a personalized display actually taking into consideration the second vehicle 300 traveling in front of the first vehicle 200.

The present invention achieves this objective.

The concept underlying the present invention is to exploit a DB database, preferably external, which is in the form of a veritable digital library comprising an image of the rear silhouette for each vehicle on the market.

To do this, the present invention provides in the example of the , which illustrates a particular and non-limiting exemplary embodiment of the present invention, the implementation of a driving aid device 100 of the first vehicle 200 which comprises means specifically adapted to exploit this database DB and determine in this base the image corresponding to the second vehicle 300.

Such a device 100 corresponds for example to a device on board the vehicle 200, for example a computer.

In the example described here, the device 100 according to the present invention is for example configured for the implementation of the steps of the method described with regard to the .

Examples of such a device 100 include, but are not limited to, on-board electronic equipment such as a vehicle's on-board computer, an electronic computer such as an ECU ("Electronic Control Unit"), a telephone smart phone, tablet, laptop. The elements of device 100, individually or in combination, can be integrated into a single integrated circuit, into several integrated circuits, and/or into discrete components. The device 100 can be produced in the form of electronic circuits or software (or computer) modules or else a combination of electronic circuits and software modules.

In this example, the device 100 comprises one (or more) processor(s) 30 configured to execute instructions for carrying out the steps of the method and/or for executing the instructions of the software or software embedded in the device 100. The processor 30 may include integrated memory, an input/output interface, and various circuits known to those skilled in the art.

The device 100 further comprises at least one memory 20 corresponding for example to a volatile and/or non-volatile memory and/or comprises a memory storage device which can comprise volatile and/or non-volatile memory, such as EEPROM, ROM , PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

The computer code of the on-board software or software comprising the instructions to be loaded and executed by the processor is for example stored on the memory 20.

According to various particular and non-limiting examples of embodiment, the device 100 is coupled in communication with other similar devices or systems and/or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French "Telematic Control Unit"), for example via a communication bus or through dedicated input/output ports.

In the example described here in , the device 100 thus comprises a block 50 of interface elements for communicating with external devices, for example a remote server or the “cloud”, a TCU unit. Block 50 interface elements include one or more of the following interfaces:

- RF radio frequency interface, for example of the Bluetooth® or Wi-Fi® type, LTE (from English "Long-Term Evolution" or in French "Evolution à long terme"), LTE-Advanced (or in French LTE-advanced );

- USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French);

- HDMI interface (from the English “High Definition Multimedia Interface”, or “Interface Multimedia Haute Definition” in French);

- LIN interface (from English “Local Interconnect Network”, or in French “Réseau interconnecté local”).

Data is for example loaded to the device 100 via the block 50 interface using a Wi-Fi® network such as according to IEEE 802.11, an ITS G5 network based on IEEE 802.11p or a mobile network such as a 4G network. (or LTE Advanced according to 3GPP release 10 – version 10) or 5G, in particular an LTE-V2X network.

The device 100 can thus communicate with external devices for example to obtain other relevant information for driving assistance such as for example geolocation data or traffic data.

This interface block 50 can also be used to update the DB database.

In the example described here in , the device 100 further comprises a communication interface 60 which makes it possible to establish communication with other devices (such as other computers of the on-board system) or the ADAS system 210 via a communication channel 61.

This communication interface 60 also makes it possible to establish communications with the database DB which here comprises a catalog updated continuously or periodically and presented in the form of a digital library comprising for each silhouette and/or each vehicle model from a determined list of vehicles an image representative of the vehicle.

This library and its operating principle is described later in the description.

The communication interface 60 corresponds for example to a transmitter configured to transmit and receive information and/or data via the communication channel 61. The communication interface 60 corresponds for example to a CAN-type wired network (of the 'English "Controller Area Network" or in French "Réseau de Contrôleurs"), CAN FD (from English "Controller Area Network Flexible Data-Rate" or in French "Réseau de Contrôleurs à Flow de Data Flexible"), FlexRay ( standardized by ISO 17458) or Ethernet (standardized by ISO/IEC 802-3).

According to a particular and non-limiting example of embodiment, the device 100 can supply output signals to one or more external devices, such as a display screen 201, touch-sensitive or not, one or more loudspeakers 202 and/or other peripherals 203 (projection system) via output interfaces 70, 80 and 90 respectively.

According to a variant, one or the other of the external devices is integrated into the device 100.

There illustrates a flowchart of the different steps of an augmented reality driving aid method for a first vehicle 200, according to a particular and non-limiting embodiment of the present invention.

The method is for example implemented by a device on board the vehicle 200, for example by one or more processors on board the vehicle 200, or by the device 100 of the .

Thus, in the example described here and illustrated in Figures 1 to 3, the sensor module 10 of the device 100 comprises a camera which captures during a step S0 an image I1 of the road environment 1.

In this example, the camera 10 is of the CMV type (for "Multifunction Video Camera").

Those skilled in the art will understand here that the integration of other sensors for the module 10 can be envisaged here, such as for example:

- one or more millimetric wave radars arranged at the front of the vehicle 200; each radar is adapted to emit electromagnetic waves and to receive the echoes of these waves sent back by one or more objects, with the aim of detecting obstacles and their distances from the vehicle and thus of recovering points representative of the silhouette of the second vehicle 300; and or

- one or more LIDAR (s) (from the English "Light Detection And Ranging", or "Detection and estimation of the distance by the light" in French) arranged at the front of the vehicle 200, a LIDAR sensor corresponding to a optoelectronic system composed of a laser transmitter device, a receiver device comprising a light collector (to collect the part of the light radiation emitted by the transmitter and reflected by any object located on the path of the light rays emitted by the transmitter ) and a photodetector which converts the collected light into an electrical signal; a LIDAR sensor thus makes it possible to detect the presence of objects situated in the light beam emitted and to measure the distance between the sensor and each object detected with the aim of recovering points representative of the silhouette of the second vehicle 300; and or

- one or more cameras (associated or not with a depth sensor) for the acquisition of one or more images of the environment around the first vehicle 200 located in the field of vision of the camera or cameras.

On reception S1 of this image I1, the processor 30 applies data processing S2 to said image I1 for recognition in the road environment 1 of the third vehicle 300.

More particularly, this processing step S2 comprises a detection in the image I1 of the presence of a second vehicle 300 in the road environment 1 then a recognition in this image I1 of identification information I300 which makes it possible to identify the second vehicle 300.

In the example described here, the identification information I300 comprises the silhouette of the vehicle 300 seen from the rear as well as the model of the vehicle 300. Other identification information I300 may be envisaged by those skilled in the art.

This information I300 is for example in the form of points in the image I1, these points together defining the outline or the silhouette of the second vehicle 300. In this example, this information I300 is directly deduced from the processing S2 carried out on the image acquired I1. The recognition step is implemented according to any method known to those skilled in the art, for example by processing the images obtained from the camera 10 by applying a segmentation of each image into objects (for example a segmentation based on color and /or the depth associated with the pixels of the image) and comparing each object with a library of positive images (i.e. images representing vehicles) and a library of negative images (i.e. i.e. non-vehicle images).

The recognition of the silhouette of the second vehicle can be done by image processing techniques making it possible to detect the contours, for example with color gradients or others.

Pattern recognition can also be done through image processing techniques to recognize characters in the image.

According to another example, the recognition of the silhouette and/or of the vehicle model can also be obtained by a so-called machine learning method, for example implemented by a neural network .

Once the silhouette and/or the model of the second vehicle 300 has been recognized, the processor 30 determines during a step S3 an image I representative of the second vehicle as a function of this information I300.

This step S3 requires the interrogation by the device 100 of the database DB (which is here in this example external to the device 100).

As indicated previously, this database DB constitutes an image catalog which includes a list of vehicles and assigns, to each vehicle of this list defined by a silhouette and/or a model, an image I representative of the rear face of that -this.

Thus, it is possible to establish a correspondence between the model and/or the silhouette of each vehicle on the market and at least one image I of a realistic view of a rear face of this vehicle.

By interrogating this database DB, the processor 30 determines during this step S3 an image I by comparing the information I300 (model/silhouette) with the database DB.

Once determined, this image I is displayed on the screen 201 of said first vehicle 200. It may be a head-up display or any other type of display on a screen 201.

It will be understood here that, in order to have an up-to-date database taking account of new vehicles on the market, regular S4 updating of the base is planned, in particular via, for example, the communication interface of block 50 which allows communication to a remote server.

The present invention also relates to a vehicle, for example automobile or more generally an autonomous land motor vehicle, comprising the device 100 of the .

Of course, the present invention is not limited to the embodiments described above but extends to a control method which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for the implementation of such a method.

Claims (10)

Method for assisting the driving in augmented reality of a first vehicle (200) traveling in a road environment (1), said method comprising the following steps:
- reception (S1) of at least one item of information (I1) representative of said road environment (1) captured by a sensor module (10) on board said first vehicle (200);
- data processing (S2) of said at least one piece of road environment information (I1) for recognition in said road environment (1) of at least one piece of identification information (I300) relating to a second vehicle (300) traveling in front of said first vehicle (200);
- determination (S3) of an image (I) representative of the second vehicle (300) as a function of said at least one piece of identification information (I300); And
- display (S4) of said image (I) representative of said second vehicle (300) on a screen (201) of said first vehicle (200). Method according to claim 1, in which the determination (S3) of said image (I) representative of said second vehicle (300) is carried out by comparing said at least one piece of identification information (I300) with a database (DB) comprising for each vehicle (200, 300) of a determined list of vehicles a predetermined image (I) representative of said vehicle (200, 300). Method according to claim 1 or 2, wherein said at least one piece of identification information (I300) relating to a second vehicle (300) comprises a silhouette and/or a model of said second vehicle (300). Method according to Claim 3 appended to Claim 2, in which the database (DB) is in the form of a digital library comprising, for each silhouette and/or each determined vehicle model, an image representative of said vehicle. Method according to claim 4, which comprises updating (S4) said database (DB). Method according to any one of the preceding claims, in which the said image (I) of the second vehicle (300) is representative of a rear face of the said second vehicle (300). A method according to any preceding claim, wherein said display of said image is of the head-up display type. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when said instructions are executed by a processor. Augmented reality driving aid device for a first vehicle traveling in a road environment, said device (100) comprising a memory (20) associated with at least one processor (30) configured for the implementation of the steps of the method according to any one of claims 1 to 7. Vehicle (200) comprising a device (100) according to claim 9.