WO2018162646A1

WO2018162646A1 - Updating a landmark map

Info

Publication number: WO2018162646A1
Application number: PCT/EP2018/055773
Authority: WO
Inventors: Kichun JO; Paulo Resende; Benazouz Bradai
Original assignee: Valeo Schalter und Sensoren GmbH
Current assignee: Valeo Schalter und Sensoren GmbH
Priority date: 2017-03-10
Filing date: 2018-03-08
Publication date: 2018-09-13
Anticipated expiration: 2019-09-10
Also published as: DE102017105086A1

Abstract

The update of landmark maps is to be improved. Thereto, travelling in an area (12) with a vehicle (10), capturing the area (12) from the vehicle (10) to obtain area data and obtaining vehicle position data of the vehicle in the area (12) are effected. A presence or absence of a landmark (14 to 19 and 20 to 27, respectively) changed with respect to the landmark map in the area data is ascertained with respect to the vehicle position data. Finally, the landmark map is updated with respect to the landmark.

Description

Updating a landmark map

The present invention relates to a method for updating a landmark map by traveling in an area with a vehicle, capturing the area from the vehicle to obtain area data, ascertaining a landmark from the area data and localizing the vehicle based on the landmark and the landmark map. Moreover, the present invention relates to an apparatus for ascertaining a change of a landmark for localizing a vehicle, wherein the apparatus comprises a sensor device for capturing an area from the vehicle to obtain area data and a localizing device for obtaining vehicle position data of the vehicle in the area. Moreover, the present invention relates to a corresponding driver assistance system as well as to a

corresponding vehicle.

Landmarks are points of orientation, by which a localization can be performed if the exact position thereof is known. Prominent buildings, but also barriers, fences, road markings, traffic lights, traffic signs, lanes and many vertical objects can serve as landmarks. Such landmarks can be geographically combined in a landmark map in order that they can be used for localization.

Generally, it is possible to determine the own position based on a landmark, which one sees or which any type of object recognition system including sensors and signal processing captures. For example, if the object recognition system detects a church and a map is available, which describes the position of the church, it is possible to determine, where one is located. This type of the self-positioning can be referred to as landmark- based self-localization. As mentioned, the map with the church and other landmarks can be referred to as landmark map. This landmark map can be stored in the vehicle or "remotely" on an external server.

A landmark-based self-localization provides a precise position and direction of the own vehicle by matching the landmarks from actual perception or from real-time captures with those from precise landmark maps. This can be used for highly automated vehicles.

Fig. 1 describes a known structure for self-localization, which is based on a landmark comparison. A sensor device 1 has one or more sensors S1 to S5 and so on. Among these sensors, there can be a front camera, a surround-view camera, a laser scanner, a radar, an ultrasonic sensor and the like. They monitor an area around the vehicle and provide corresponding area data 2. An analyzing device 3 analyzes the area data 2 and obtains landmarks L1 to L7 and so on therefrom. Such a landmark can for example be a lane, a traffic sign, a traffic light, a road marking, a barrier and so on. Corresponding landmark data 4 is provided to a comparing device 5, which compares the actually captured landmarks L1 to L7 to mapped landmarks L1 ' to L7' and so on. These mapped landmarks L1 ' to L7' and so on originate from a database 6, which is internal or external to the own vehicle. Thus, a landmark map can be specifically locally stored in the vehicle and/or a landmark map is on an external server, which is accessible via wireless communication. The database 6 provides a corresponding signal 7 of the mapped landmarks L1 ' to L7' and so on to the comparing device 5.

Due to the comparison of the actually captured landmarks L1 ... and the mapped landmarks L1 ' the comparing device 5 provides a corresponding comparison signal 8 to a localizing device 9. It ascertains the position and/or orientation of the own vehicle 10 based on the comparison signal 8 and provides a corresponding position signal 1 1 . In this manner, the position and/or direction of the own vehicle 10 can be updated.

The precise landmark map in the database 6 can be provided by a provider. The map provider generates the precise map with its own special mapping system. Therein, the landmarks can be centimeter-precisely indicated. However, if a landmark changes, this can decrease the accuracy of the self-localization since a discrepancy between the perception of the sensor device and the precise landmark map of the database 6 occurs. Such a discrepancy can arise if for example new traffic lights are installed, lanes are removed or the positions of traffic signs are changed.

From the printed matter DE 10 2013 21 1 696 A1 , a method for completing or updating a digial road map of a geographic region is known. A digital image of a course section is captured by means of a camera of a vehicle. Furthermore, the current position of the motor vehicle in width direction of the course section or a characteristic of the course section is ascertained based on the image by means of a computing device of the motor vehicle. The geographic position is communicated to a server device, which updates the digital road map.

Moreover, the printed matter WO 2015/155133 A1 discloses a system for autonomous vehicle guidance, which is formed such that at least restricted or temporally limited vehicle guidance is still possible if a part of the components of the system fails. Landmarks such as guardrails, guide posts or traffic signs are recognized by a camera, which allows particularly accurate positioning. This in particular applies if the accurate sites of the landmarks are known to the system. Thereto, a suitable map or database can be present.

Thus, the object of the present invention is in improving the update of landmark maps. Thereto, a corresponding method as well as a corresponding apparatus is to be provided.

According to the invention, this object is solved by a method as well as an apparatus according to the independent claims. Advantageous developments of the invention are apparent from the dependent claims.

Corresponding to the present invention, accordingly, a method for updating a landmark map is provided. Preferably, a landmark map corresponds to a digital map, in which the positions of landmarks are registered. In order to now update such a landmark map, an actual area, for example a road, a place or the like, is traveled with the vehicle. The area is captured from the vehicle to obtain area data. Hereto, sensors internal to vehicle are preferably employed, such as for example a camera of the vehicle. In the area, vehicle position data of the vehicle is obtained. The position of the vehicle can for example be ascertained by GPS data. This thus obtained vehicle position data serves for associating the captured area with an actual geographic position.

Now, ascertaining a presence or absence of a landmark changed with respect to the landmark map in the area data is advantageously effected based on the vehicle position data. Thus, the area data is analyzed with respect to landmarks. If the presence or absence of a landmark in the area data with respect to the vehicle position data is determined, thus, the landmark map can be updated with respect to this landmark. If the presence or absence of the landmark does not have changed with respect to the present landmark map, thus, update of the landmark map is not required. Thus, it is

advantageously taken into account if the presence or absence of a landmark has changed. Only such a change requires the update of the landmark map. In this manner, the process of updating the landmark map can be simplified and the corresponding data amount can be reduced.

Preferably, in ascertaining the presence or absence changed with respect to the landmark map, a corresponding classification is performed. Thus, a landmark is classified as present if it is actually present in the area data captured from the vehicle. If a landmark is not found in the area data, but is registered in the landmark map, thus, this landmark not found in reality is classified as absent. Such a classification facilitates the further data processing.

In a development, the capture of the area is effected by one or more sensors of the vehicle, in particular a camera, a radar and/or a lidar. Other sensors such as ultrasonic sensors, laser scanners and the like can of course also be employed for capturing the area. It is only essential that the one or more sensors are capable of collecting suitable (image) data about landmarks. Thereby, current data about the area traveled by the vehicle can be collected.

As was already mentioned above, the vehicle position data can be ascertained by a GPS system. Alternatively, the landmark or a further landmark can be ascertained from the area data and compared to the landmark map for obtaining the vehicle position data of the vehicle. Thus, the vehicle is localized based on one or more landmarks. As appropriate, a coarse localization by means of a GPS system and a fine localization with the aid of the one or the more landmarks can be effected. This means that any combination of localization methods is possible.

In a preferred configuration, the and/or the further landmark represent at least one from the group of lane, traffic light, traffic sign, roadway marking and curb. However, a landmark can also be any vertical object, such as for example a building, a tree, a fence, a bridge or the like as initially mentioned. The landmarks immediately on or at the road are suitable for localizing vehicles in road traffic in particular manner. Therein, landmarks, which are only in low distance from the vehicle, are particularly valuable such that they can be well captured from the vehicle and be used for highly accurate localization.

In an advantageous configuration, the position data includes a position and an orientation of the vehicle. In order to be able to locate the landmarks from the vehicle, the accurate orientation of the vehicle is possibly required. If the orientation of the vehicle is not known, at least one further landmark and the vehicle position have to be present to be able to perform an accurate localization of the landmarks.

Principally, the update of the landmark map can be performed in the vehicle. Alternatively, the ascertained presence or absence of the landmark changed with respect to the landmark map can be communicated to a map provider. Thus, the map provider can be provided with information whether or not an individual landmark still exists. He can also obtain information about the fact if additional landmarks have appeared. However, the update does not have to be individually performed for each landmark. Rather, an updated landmark map with respect to the captured area, in which the vehicle is located, can also be sent to the map provider. Thereby, a plurality of landmarks can possibly be updated at the provider with a single data communication.

In a development of the method according to the invention, the travel in the area with the vehicle is autonomously effected. This means that the vehicle is at least currently not controlled by a driver, but in automatic manner. Since the environmental region of the vehicle is captured and registered, respectively, in autonomous driving anyway, the landmarks captured therein can also be used to update a landmark map in the vehicle or at a provider.

The registered or non-registered landmarks can be correspondingly classified. In the classification, besides the classes of the changed presence and the changed absence with respect to a present landmark map, the class of an unchanged presence and the class of a false ascertainment can further also be used. In this manner, any analysis result can be associated with one of four classes. A first class relates to a landmark, which is recorded in the landmark map and is also actually detected. A second class relates to landmarks, which are not recorded in the landmark map and which are newly captured. A third class relates to landmarks, which are registered in the landmark map, but which do no longer exist in reality and thus either are not captured by the sensor system of the vehicle. Finally, a fourth class relates to landmarks, which are not contained in the landmark map and are only temporarily detected by the sensor system of the vehicle (false detection). Such a classification can be reliably employed for updating landmark maps.

The update of the landmark map can be effected in the vehicle, wherein an external landmark map is additionally used thereto, which is loaded by the vehicle via wireless communication. Thus, a landmark map is loaded into the vehicle and there updated based on the landmarks captured around the vehicle. As appropriate, the thus updated landmark map can be again communicated to a provider or another recipient via a wireless communication link.

Particularly preferably, the method according to the invention for updating a landmark map can be used for a driver assistance system. Many driver assistance systems are dependent on accurate positioning of the vehicle. Therefore, correspondingly actual landmark maps allowing a very exact positioning and/or orientation determination of the vehicle are particularly valuable for driver assistance systems. Thus, it is also

advantageous if the driver assistance systems themselves can update their landmark maps in the mentioned manner.

According to the invention, the above mentioned object is also solved by an apparatus for ascertaining a change of a landmark for localizing a vehicle, wherein the apparatus comprises:

- a sensor device for capturing an area from the vehicle to obtain area data, and

a localizing device for obtaining vehicle position data of the vehicle in the area,

- wherein the analyzing device of the apparatus is formed for ascertaining a presence or absence of a landmark changed with respect to the landmark map in the area data with respect to the vehicle position data, and

- a data processing device of the apparatus is formed for updating the landmark map with respect to the landmark or for providing corresponding update data.

The possibilities of variation and advantages presented above in context with the method according to the invention analogously also apply to the apparatus according to the invention for ascertaining a change of a landmark, which can be used for localizing a vehicle. The explained methods features are to be regarded as functional features of the apparatus in this case.

As was already illustrated, a vehicle can be equipped with such an apparatus.

Alternatively, a vehicle can also be provided with the mentioned driver assistance system.

The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations without departing from the scope of the invention. Thus, implementations are also to be considered as encompassed and disclosed by the invention, which are not explicitly shown in the figures and explained, but arise from and can be generated by separated feature combinations from the explained implementations. Implementations and feature combinations are also to be considered as disclosed, which thus do not have all of the features of an originally formulated independent claim.

Moreover, implementations and feature combinations are to be considered as disclosed, in particular by the implementations set out above, which extend beyond or deviate from the feature combinations set out in the relations of the claims.

In the attached figures, there show:

Fig. 1 a precise self-localization based on multiple landmark matches between an accurate landmark map and landmarks captured from the vehicle;

Fig. 2 an exemplary classification of landmarks in the area of a vehicle; and

Fig. 3 an example of a precise localization based on a selective measurement update with the aid of a completed landmark map.

The embodiments explained in more detail below present preferred embodiments of the present invention.

Vehicles and driver assistance systems, respectively, operate for self-localization often with landmark maps. It is necessary to always keep them up to date. Hereto, an online localization and map management system for highly automated vehicles can for example be provided based on a landmark matching classifier.

For example, the landmark matching classifier can classify landmark matching operations into four types. These types are presented in more detail in context with Fig. 2 and 3. Based on the classified type, an online localization algorithm can estimate the own position (possibly including orientation). Thereby, the map management system can generate a temporary landmark map and report changes in the landmark map to a map provider.

In a specific example, the vehicle 10 travels in an area 12. The vehicle 10 for example has a camera for capturing images and video data of the area 12, respectively. These images and video data, respectively, represent area data of the area 12.

The vehicle 10 localizes itself and therein obtains vehicle position data. This localization is for example effected by means of a GPS system integrated in the vehicle 10 and/or based on landmarks in the area 12, which is captured from the vehicle 10. The vehicle 10 has a landmark map, in which landmarks 13 to 19 are registered. It is for example a digital landmark map, which is stored in a memory of the vehicle 10 and/or is obtained online from a server. The landmark map usually does not only show the positions of the landmarks, but also the type thereof and possibly the extensions thereof.

The vehicle 10 captures the area 12 by its sensor system (in particular one or more cameras) and a suitable analyzing algorithm or a corresponding analyzing device extracts the currently actually present landmarks 20 to 27 from the area data obtained by the vehicle 10. To these captured landmarks 20 to 27, the analyzing algorithm possibly obtains the position thereof and optionally also further data such as type, extension et cetera.

Now, in order to be able to perform an update of the landmark map, the landmarks 14 to 19, which are in the map area of interest, ascertained based on the vehicle position data, are compared to the actually ascertained landmarks 20 to 27. Therein, only that map section from the landmark map is preferably compared, which corresponds to the area 12 captured by the sensor system of the vehicle 10. Within the scope of this comparison, matching of the mapped landmarks 14 to 19 and of the actually captured landmarks 20 to 27 can be effected. For this matching, classification of the individual landmark comparison operations is favorably performed. An analyzing device or matching device can perform this matching of the landmarks in the vehicle 10.

For this classification of the landmark comparison operations, four classes can be provided: a) class "normal": A landmark comparison operation or a landmark is classified into this class if the respective landmark exists in the precise landmark map and is also detected by the sensor system of the vehicle 10. b) class "new": A landmark comparison operation or a landmark is classified into this class if it is not present in the precise landmark map, but is continuously detected by the sensor system of the vehicle 10. c) class "deleted": A landmark comparison operation or a landmark is classified into this class if the corresponding landmark is present in the precise landmark map, but is not detected by the sensor system of the vehicle 10. d) class "outlier": A landmark comparison operation or a landmark is classified into this class if its corresponding landmark is not present on the precise landmark map, but is temporarily detected by the sensor system of the vehicle 10.

In the example of Fig. 2, the landmarks 21 to 25 can be associated with the class "normal" since the corresponding landmarks 14 to 18 also exist in the landmark map. The landmark 26 would come into the class "new" since its counterpart is absent in the landmark map. In contrast, the landmark 19 of the landmark map would come into the class "deleted" since a landmark was no longer perceived by the sensor system of the vehicle 10 there. In contrast, the two detected landmarks 20 and 27 would come into the class "outlier" since they were only temporarily detected and corresponding counterparts do not exist in the landmark map. With the aid of this classification, it is possible to perform a simple update of the landmark map. In particular, only the two landmarks 19 and 26 have to be correspondingly changed. The remaining, already existing landmarks are indirectly confirmed hereby. An update of the landmark 13 is not effected because it is outside of the range of perception 12 of the sensor system of the vehicle 10.

Below, an online localization and a map management based on the classification of the landmark comparison operations are illustrated in more detail. Hereto, Fig. 3 presents an overall architecture of the online localization and of the map management system based on a corresponding classifier. After associating the mapped and detected landmarks with each other, the classifier classifies the landmark comparison operations into the four types: "normal", "new", "deleted" and "outlier".

Corresponding to the architecture illustrated in Fig. 3, a sensor device 1 and an analyzing device 3 are provided like in the architecture of Fig. 1 . Reference is made to the description of Fig. 1 . This also applies to those elements, which are provided with the same reference characters as in Fig. 1 . Especially, the sensor device 1 therefore has a plurality of sensors S1 to S5 and so on. The area data 2 generated thereby is passed to the analyzing device 3, which extracts one or more landmarks L1 to L7 and so on therefrom.

The comparing device 5 compares the detected landmarks L1 to L7 and so on to mapped landmarks L1 ' to L7' and so on. The respectively resulting result of comparison is subjected to a classification K. The above mentioned classes "normal" no, "new" ne, "deleted" ge and "outlier" ou are available for the classification. A data processing or localizing device 9 receives the landmarks of the class "normal" no and therewith executes a usual updating process for the self-localization of the vehicle 10 and provides a corresponding position signal 1 1 to the vehicle 10.

For landmarks of the class "new" ne, a partial online SLAM algorithm 28 (Simultaneously localization and mapping) can be performed, in which simultaneous localization and mapping can be effected. This algorithm is applied to improve the estimation quality for the vehicle position data and to create a new landmark map or new mapped landmarks 29 at the same time. After generating the new mapped landmarks 29, the corresponding map information is communicated from a map provider 30 or a cloud for automated driving to inform about the newly generated landmarks. For the partial online SLAM method 28, so- called Rao-Blackwellzied particle filters (RBPFs), such as for example a FastSLAM algorithm and an UFastSLAM algorithm, can be employed because the localization and the mapping process are at least theoretically independent of each other. The difference between a normal online SLAM and a partial online SLAM is in that the latter uses the precise landmark map of a map provider and the local landmark map at the same time for matching the landmarks.

The information of the landmarks from the class "deleted" is not used to update the localization of the vehicle. This information is only provided to the map provider to inform about the absence of the concerned landmark. Landmarks of the class "outlier" remain completely unused, i.e. their information is ignored.

The landmark comparison classifier for classifying the landmark comparisons or the landmarks can operate based on two approaches: Classifiers based on monitored learning and classifiers based on statistic distance. The monitored learning is machine learning, wherein the training is based on likelihoods of landmark matches. Support vector machines, neuronal networks, decision trees and the like can be used. In the classification based on statistic distance, a statistic test is performed for each measurement and the Mahalanobis distance can for example be ascertained.

With the presented apparatus and the presented method, respectively, thus, a landmark map can be comfortably updated with low process effort if a vehicle detects a new or a deleted landmark.

Claims

1 . Method for updating a landmark map by

- traveling in an area (12) with a vehicle (10),

- capturing the area (12) from the vehicle (10) to obtain area data (2), and

- obtaining vehicle position data (1 1 ) of the vehicle (10) in the area (12), characterized by

- ascertaining a presence or absence of a landmark (14 to 27) changed with respect to the landmark map in the area data with respect to the vehicle position data, and

- updating the landmark map with respect to the landmark (14 to 27).

2. Method according to claim 1 ,

characterized in that

in ascertaining the presence or absence changed with respect to the landmark map, a corresponding classification (K) is effected.

3. Method according to claim 1 or 2,

characterized in that

the capture of the area (12) is effected by one or more sensors (S1 to S5) of the vehicle (10), in particular a camera, a radar and/or a lidar.

4. Method according to any one of the preceding claims,

characterized in that

for obtaining the vehicle position data (1 1 ) of the vehicle (10), the landmark (14 to 27) or a further landmark is ascertained from the area data (2) and compared to the landmark map.

5. Method according to any one of the preceding claims,

characterized in that the and/or the further landmark represent at least one from the group of lane, traffic light, traffic sign, roadway marking and curb.

6. Method according to any one of the preceding claims,

characterized in that

the vehicle position data (1 1 ) includes a position and an orientation of the vehicle (10).

7. Method according to any one of the preceding claims,

characterized in that

the ascertained presence or absence of the landmark (14 to 27) changed with respect to the landmark map is communicated to a map provider (30).

8. Method according to any one of the preceding claims,

characterized in that

traveling in the area (12) with the vehicle (10) is autonomously effected.

9. Method according to any one of claims 2 to 7,

characterized in that

in the classification (K), besides the classes of the changed presence (ne) and the changed absence (de), the class of an unchanged presence (no) and the class of a false ascertainment (ou) are also used.

10. Method according to any one of the preceding claims,

characterized in that

the update of the landmark map is effected in the vehicle (10) and an external landmark map is additionally used thereto, which is loaded by the vehicle (10) via a wireless communication.

1 1 . Driver assistance system, which is formed to execute a method according to any one of the preceding claims.

12. Apparatus for ascertaining a change of a landmark (14 to 27) for localizing a vehicle (10), wherein the apparatus comprises: - a sensor device (1 ) for capturing an area (12) from the vehicle (10) to obtain area data (2), and

- a localizing device (9) for obtaining vehicle position data (1 1 ) of the vehicle

(10) in the area (12),

characterized in that

- the analyzing device (3) of the apparatus is formed for ascertaining a

presence or absence of a landmark (14 to 27) changed with respect to the landmark map in the area data (2) with respect to the vehicle position data

(1 1 ) , and

- a data processing device of the apparatus is formed for updating the

landmark map with respect to the landmark (14 to 27) or for providing corresponding update data (29).

13. Vehicle with a driver assistance system according to claim 1 1 or an apparatus according to claim 12.