DE102024000796A1 - Method for providing map data for motor vehicles and system - Google Patents

Abstract

The invention relates to a method for providing map data (A) of a traffic scene for motor vehicles (10), in which a past movement of the motor vehicle (10), aerial photographs (14) and information from agents (12) surrounding the motor vehicle (10) are compared with one another to evaluate the map data (A), a discriminator (2) being used which has an encoder-decoder architecture, an encoder (3) converting the past movement of all detected surrounding agents (12) and the movement (X1*, X2, X3, X4) of the motor vehicle (10) into a latent encoding and converting the current aerial photograph (14) into another latent encoding, these latent encodings being merged to create a latent encoding per agent (12) with information about the aerial photograph (14), a graph neural network being used to transmit information between the agents (12) and to create a feature vector for the entire traffic scene, and a decoder (4) is used to classify whether the movements match the aerial photograph (14). The invention further relates to a system.

Description

The invention relates to a method for providing map data for motor vehicles according to patent claim 1 and to a system according to the invention.

The behavior planning of automated vehicles and the prediction of vehicles participating in traffic are essential components of autonomous vehicles.

The object of the invention is to at least partially improve the consistency of data for traffic maps.

This object is achieved by means of a method having the features of patent claim 1 and by means of a system according to the invention. Advantageous embodiments and further developments are the subject of the dependent claims.

One aspect of the invention relates to a method for providing map data for motor vehicles, in particular at least partially autonomous motor vehicles, in which a past movement of the motor vehicle, aerial photographs and information from agents in the vicinity of the motor vehicle are compared with one another in order to evaluate the map data and thus a traffic scene. Agents are objects or moving objects that enable at least one data acquisition and/or whose data and/or physical data are applicable to the method.

Here, it is envisaged that a discriminator is used that has an encoder-decoder architecture, where the encoder converts the past movement of all detected surrounding agents and the movement of the motor vehicle into a first latent encoding and converts the current aerial photograph into another second latent encoding. These latent encodings are fused to create a latent encoding per agent with information about the aerial photograph. A graph neural network is then used to transfer information between the agents and create a feature vector for the entire traffic scene, thereby modeling and/or taking into account relationships and dependencies between the different agents in the traffic scene to be evaluated. Finally, a decoder is applied to classify whether the movements match the aerial photograph and thus evaluate and/or confirm the map data.

In other words, the aim is to provide a way to create map data using secure data.

In the state of the art, learning-based methods for behavior planning and prediction have proven to be particularly accurate. This means that learning-based approaches dominate on various benchmarks. A core element of such approaches is the inclusion of the static infrastructure. The static infrastructure is usually represented by so-called high definition maps or HD maps. These HD maps are complex to obtain: A fleet of vehicles must take pictures of roads, for example using several sensors, then algorithms are used to create a first version of the HD map, and finally a human correction must be carried out. This is expensive and time-consuming.

An alternative is the use of so-called “aerial images”. Aerial photographs can be taken by satellites, airplanes, drones or other flying objects. Aerial photographs are inexpensive and available globally.

Accordingly, it is possible to create a map using aerial photographs. For a given traffic scene, a part of the map is cut out. This section only shows an area of, for example, 100×100 meters around the automated vehicle. For example, the location where the automated vehicle is currently located is centrally located in this section. The section is also processed, for example, by a learning-based planning or prediction model. For this purpose, CNNs or convolutional neural networks or vision transformers can be used, for example.

However, aerial photographs do not always show the current and/or correct data and thus also the correct roads. For example, a traffic scene may take place in a tunnel. In this case, the aerial photograph only shows a mountain. For example, a traffic scene may also take place in an area where the actual road on which the automated vehicle or motor vehicle is driving is obscured by other roads that run vertically above it. In this second example, the learning-based model may misinterpret the obscuring roads and use data or information from an incorrectly assigned road for any planning or prediction tasks and thus for providing map data.

Accordingly, the intention is to provide a method and thus, for example, to create a model that is able to independently recognize that the aerial photograph is not suitable and then forwards this information to the planning or prediction component. In other words, a method for providing map data for motor vehicles is provided, which also checks this map data.

The solution approach developed for this purpose is based on a discriminator, which solves the classification problem by checking the past movement of the automated vehicle or motor vehicle and surrounding agents recognized by perception, such as vehicles or motor vehicles, pedestrians, etc., with the extracted aerial photographs.

According to the invention, the discriminator uses an encoder-decoder architecture.

In the encoder component, the past movements of all recorded surrounding agents as well as the automated vehicle or motor vehicle are encoded by a time series encoder. The result is a latent encoding per agent in the traffic scene. In parallel, the current aerial photograph is encoded using an image encoder. These two encodings are merged, whereby various methods such as concatenation or cross-attention can be applied. This creates another latent encoding per agent, which already contains information about the aerial photograph. A graph is then created, for example, in which each agent is represented as a node. The characteristics of the nodes are the latent agent characteristics. A graph neural network performs message passing to enable the exchange of information between the nodes. A readout operator aggregates the characteristics of the nodes into a single feature vector that describes the entire traffic scene.

The decoder applies a classification to the resulting feature vector to decide whether the encoded motions match the aerial photographs, using a sigmoid activation function to produce a probability as output.

The ability of the model to perform this classification depends crucially on the training process and the dataset used.

The dataset used consists of pairs of input data and output labels. The input data includes the agents detected in the environment, the past movement of the automated vehicle and the aerial photograph. The output label is binary, for example. It states whether the detected agents and the automated vehicle are consistent with the aerial photograph, which results in the value 1 being output or not, for example, and thus the value 0 being output.

The dataset is created by automatically creating aerial photographs of some traffic scenes. The past movement of the agents and the automated vehicle or motor vehicle is depicted in this aerial photograph, in particular by using the same coordinate system.

The traffic scenes are manually checked to see if the movement fits sensibly with the roads visible in the aerial photographs. This visual test is carried out because everything has already been transferred to the same coordinate system. This manual check creates the labels, with 1 being output for consistent and 0 for inconsistent.

One problem here is that there will probably be many more positive examples (consistent) than negative examples (inconsistent). This is because street occlusion rarely occurs.

Therefore, artificial negative examples can be created, especially without manual effort. For this purpose, movements of the agents and the automated vehicle are randomly combined with an aerial photograph from another region. The traffic scene is always automatically labeled "0" as a negative example.

The encoder-decoder model or discriminator mentioned above is then trained with the created data set. Machine learning methods are used for this. The weights in the encoder and decoder are adjusted using backpropagation. The loss function used is a binary classification error function such as Binary Cross Entropy.

The discriminator is then able to classify whether the past movement of the automated vehicle and the surrounding agents detected by perception match the extracted aerial image.

This information is then used for a learning-based planner or predictor, which also knows the aerial image as input. The aim is to provide the ability to know when these known inputs are useless in order to avoid applying or extracting incorrect features, e.g. Predictions in wrong directions, on wrong tracks, etc.

As soon as the discriminator outputs a probability of less than 50%, the aerial photograph is masked. This means that every pixel receives the value "0". Otherwise, the aerial photograph is adopted unchanged. In addition, a binary "0" or "1" is also passed directly to the learning-based planner or predictor in the encoder, depending on whether the discriminator outputs a probability of less than 50% or more than 50%.

In summary, the invention includes training a discriminator that is specialized in determining whether the past movement of the automated vehicle and the surrounding perceptually detected agents such as vehicles and pedestrians match the extracted aerial image. The output of this discriminator can then be used to improve learning-based planning or prediction models.

The advantages of this method are manifold, e.g. by checking the aerial images and avoiding the extraction of false features, planning and prediction are improved. This is done with minimal labeling effort, as only visual inspection is required and negative examples can be generated automatically. Training is simple, and the discriminator can potentially even be used for several modules at the same time. In addition, existing learning-based planning and prediction models can be reused with only minor adjustments.

Further advantages, features and details of the invention emerge from the following description of a preferred embodiment and from the drawing. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and/or shown alone in the single figure(s) can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the invention.

• 1 zeigt ein System zur Bereitstellung von Kartendaten einer Karte für einen Kraftwagen.

Showing:

• 1 shows a system for providing map data of a map for a motor vehicle.

In the figure, identical or functionally equivalent elements are provided with the same reference symbols.

1 shows a system for providing map data A of a map for motor vehicles 10, in particular at least partially autonomous motor vehicles on a trajectory T, wherein a road Tx runs above the trajectory T, which is not so reliably recognizable by map data A. Accordingly, these map data A from the air and thus aerial photographs 14 are to be compared with information from agents 12 along the trajectory T.

The system has a central electronic computing device 1, which is designed to integrate and process various data sources. It is provided that entities of the system such as the motor vehicle 10, the agents 12 or other data sources are coupled to the electronic computing device 1 in a data-transmitting manner, in particular wirelessly. The central electronic computing device 1 enables the evaluation of the map data A by comparing a past movement of the motor vehicle 10, aerial photographs 14 and information from agents 12 in the vicinity of the motor vehicle 10.

The system comprises a discriminator 2 having an encoder-decoder architecture. An encoder 3 can convert the past movement of all detected surrounding agents 12 and the movement of the motor vehicle 10 into a latent encoding. Likewise, the encoder 3 can convert the current aerial photograph into another latent encoding. These latent encodings can be fused to create a latent encoding per agent 12 with information about the aerial photograph 14.

The system also includes a graph neural network that is used to transfer information between the agents 12 and to create a feature vector for the entire traffic scene. The graph neural network makes it possible to model and take into account the relationships and dependencies between the different agents 12 in the traffic scene.

Finally, the system comprises a decoder 4, which is designed to classify whether the movements X1, X2, X3, X4 of the agents 12 match the aerial photograph 14. The decoder 4 plays a crucial role in confirming the map data and enables the required consistency and reliability of the recorded data. In the 1 For example, the movement X1* is shown as a potential error that could pose a problem for the motor vehicle 10, since a curve is approaching that is not immediately recognizable due to the road Tx. Accordingly, it is intended to compare this map data A with possible information from the agents 12 and to ensure that they match. At the latest at time t, the motor vehicle 10 the information is transmitted that driving straight ahead is no longer possible.

In summary, the invention proposes a method for using partially obscured satellite maps by means of a discriminator 2.

List of reference symbols

1

electronic computing device

2

Discriminator

3

Encoders

4

decoder

10

Motor vehicles

12

agent

14

Aerial photography

t

time

T

Trajectory

Tx

Street

X1, X2, X3, X4

Movements

X1*

Movement

A

Map data

Claims (6)

Method for providing map data (A) of a traffic scene for motor vehicles (10), in which a past movement of the motor vehicle (10), aerial photographs (14) and information from agents (12) surrounding the motor vehicle (10) are compared with one another to evaluate the map data (A), whereby: - a discriminator (2) is used which has an encoder-decoder architecture, whereby an encoder (3) converts the past movement of all detected surrounding agents (12) and the movement (X1*, X2, X3, X4) of the motor vehicle (10) into a latent encoding and converts the current aerial photograph (14) into another latent encoding, - these latent encodings are merged to create a latent encoding per agent (12) with information about the aerial photograph (14), - a graph neural network is used to transmit information between the agents (12) and to generate a feature vector for the entire traffic scene, and - a decoder (4) is applied to classify whether the movements match the aerial photograph (14). Procedure according to Claim 1 , characterized in that a training process is used with a data set of input data containing the detected agents (12), the past movement of the motor vehicle (10) and the aerial photograph (14), wherein the output label is binary and indicates whether the detected agents and the motor vehicle (10) are consistent with the aerial photograph (14) or not, wherein the data set contains both manually created positives and artificially created negatives. Procedure according to Claim 1 or 2 , characterized in that the trained discriminator (2) is used to evaluate the aerial photograph (14) for a learning-based planner or predictor. Method according to one of the preceding claims, characterized in that the learning-based planner or predictor is provided with the assessment of the static infrastructure based on the aerial photograph (14) in order to detect whether the aerial photograph is unusable. System for providing map data (A) of a traffic scene for motor vehicles (10), in which, in order to evaluate the map data (A), a past movement (X1*, X2, X3, X4) of the motor vehicle, aerial photographs (14) and information from agents in the vicinity of the motor vehicle (10) are compared with one another by means of a central electronic computing device (1), whereby: - a discriminator (2) can be used which has an encoder-decoder architecture, whereby the past movement of all detected surrounding agents and the movement of the motor vehicle can be converted into a latent encoding and the current aerial photograph can be converted into another latent encoding by means of the encoder (3), - these latent encodings can be merged in order to create a latent encoding per agent with information about the aerial photograph, - a graph neural network can be used to transfer information between the agents and to create a feature vector for the entire traffic scene, and - a decoder (4) is applicable to classify whether the movements (X1*, X1, X2, X3, X4) match the aerial photograph (14). System according to Claim 5 , characterized in that all information for the provision of map data (A) is stored in a memory coupled to the electronic computing device (1) with access for other fleet vehicles.

Images