WO2021056185A1

WO2021056185A1 - Systems and methods for partially updating high-definition map based on sensor data matching

Info

Publication number: WO2021056185A1
Application number: PCT/CN2019/107522
Authority: WO
Inventors: Yanke Wang
Original assignee: Beijing Didi Infinity Technology and Development Co Ltd
Current assignee: Beijing Didi Infinity Technology and Development Co Ltd
Priority date: 2019-09-24
Filing date: 2019-09-24
Publication date: 2021-04-01
Anticipated expiration: 2022-03-24

Abstract

A method for updating an HD map is provided. The method may include receiving, by a communication interface, the HD map and segmenting, by at least one processor, the HD map into a plurality of segments. The method may further include receiving, by the communication interface, sensor data of at least one landmark within a segment acquired by at least one sensor and matching, by the at least one processor, the sensor data with corresponding landmark data associated with the HD map. The method may also include determining, by the at least one processor, that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold and updating the segment of the HD map by updating the corresponding landmark data.

Description

SYSTEMS AND METHODS FOR PARTIALLY UPDATING A HIGH-DEFINITION MAP BASED ON SENSOR DATA MATCHING

TECHNICAL FIELD

The present disclosure relates to systems and methods for updating a high-definition (HD) map, and more particularly to, systems and methods for updating an HD map based on sensor data matching using a neural network.

BACKGROUND

Autonomous driving technology relies heavily on an accurate map. For example, accuracy of a navigation map is critical to functions of autonomous driving vehicles, such as positioning, ambience recognition, decision making and control. HD maps may be obtained by aggregating images and information acquired by various sensors, detectors, and other devices equipped on vehicles as they drive around. For example, a vehicle may be equipped with multiple integrated sensors such as a LiDAR and one or more cameras, to capture features of the road on which the vehicle is driving or the surrounding objects.

Due to re-planning, new developments, constructions, and other infrastructure changes, HD maps need to be updated routinely in order to accurately reflect the road information. For example, a single-lane road may be expanded to a two-lane road, and accordingly, the road marks, traffic signs and traffic lights may change or move. Whenever a change happened, typically the whole HD map would be updated entirely to reflect the change. This can be time consuming and inefficient. For example, the data of the unchanged part would also be processed again. Therefore, an improved system and method is needed for updating just the portion of an HD map that is actually changed.

Embodiments of the disclosure address the above problems by providing methods and systems for partially updating an HD map based on sensor data matching using a neural network.

SUMMARY

Embodiments of the disclosure provide a method for updating an HD map. The method may include receiving, by a communication interface, the HD map and segmenting, by at least one processor, the HD map into a plurality of segments. The method may further include receiving, by the communication interface, sensor data of at least one landmark within a segment acquired by at least one sensor and matching, by the at least one processor, the sensor data with corresponding landmark data associated with the HD map. The method may also include determining, by the at least one processor, that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold and updating, by the at least one processor, the segment of the HD map by updating the corresponding landmark data.

Embodiments of the disclosure also provide a system for updating an HD map. The system may include a communication interface configured to receive the HD map and sensor data of at least one landmark within a segment acquired by at least one sensor. The system may further include a storage configured to store the HD map and the sensor data. The system may also include at least one processor. The at least one processor may be configured to segment the HD map into a plurality of segments and match the sensor data with corresponding landmark data associated with the HD map. The at least one processor may be further configured to determine that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold and update the segment of the HD map by updating the corresponding landmark data.

Embodiments of the disclosure further provide a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method for updating an HD map. The method may include receiving the HD map and segmenting the HD map into a plurality of segments. The method may further include receiving sensor data of at least one landmark within a segment acquired by at least one sensor and matching the sensor data with corresponding landmark data associated with the HD map. The method may also include determining that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold and updating the segment of the HD map by updating the corresponding landmark data.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic diagram of an exemplary vehicle equipped with sensors, according to embodiments of the disclosure.

FIG. 2 illustrates a block diagram of an exemplary system for updating an HD map, according to embodiments of the disclosure.

FIG. 3 illustrates a flowchart of an exemplary method for updating an HD map, according to embodiments of the disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

FIG. 1 illustrates a schematic diagram of an exemplary vehicle 100 having a plurality of sensors 140 and 150, according to embodiments of the disclosure. Consistent with some embodiments, vehicle 100 may be a survey vehicle configured for acquiring data for constructing and/or updating an HD map or three-dimensional (3-D) city modeling.

In some embodiments, the HD map may be originally constructed using sensor data acquired by sensor 140 (e.g., a LiDAR) and sensor 150 (e.g., one or more cameras) . In some embodiments, sensor 140 may be a LiDAR. LiDAR measures distance to a target by illuminating the target with pulsed laser light and measuring the reflected pulses with a sensor. Differences of the time for laser light sending and returning, and wavelengths can then be used to make digital three-dimensional (3-D) representations of the target. The sensor data acquired by sensor 140 may include e.g., point cloud data.

In some embodiments, sensor 150 may be mobile terminals configured to capture images. For example, sensor 150 may include one or more cameras or other cost-effective imaging devices such as a monocular, binocular, or panorama camera that may acquire a plurality of images (each known as an image frame) as vehicle 100 moves along a trajectory.

In some embodiments, sensor 140/150 may be equipped, mounted, or otherwise attached to vehicle 100 (e.g., through mounting structure 130) , such that sensor 140/150 may be carried around by vehicle 100. Mounting structure 130 may be an electro-mechanical device installed or otherwise attached to body 110 of vehicle 100. In some embodiments, mounting structure 130 may use screws, adhesives, or another mounting mechanism. Vehicle 100 may be additionally equipped with sensor 140/150 inside or outside body 110 using any suitable mounting mechanisms. It is contemplated that the manners in which sensor 140/150 can be equipped on vehicle 100 are not limited by the example shown in FIG. 1 and may be modified depending on the types of sensor 140/150 and/or vehicle 100 to achieve desirable sensing performance.

It is contemplated that vehicle 100 may be an electric vehicle, a fuel cell vehicle, a hybrid vehicle, or a conventional internal combustion engine vehicle. Vehicle 100 may have a body 110 and at least one wheel 120. Body 110 may be any body style, such as a sports vehicle, a coupe, a sedan, a pick-up truck, a station wagon, a sports utility vehicle (SUV) , a minivan, or a conversion van. In some embodiments, vehicle 100 may include a pair of front wheels and a pair of rear wheels, as illustrated in FIG. 1. However, it is contemplated that vehicle 100 may have more or less wheels or equivalent structures that enable vehicle 100 to move around. Vehicle 100 may be configured to be all wheel drive (AWD) , front wheel drive (FWR) , or rear wheel drive (RWD) . In some embodiments, vehicle 100 may be configured to be operated by an operator occupying the vehicle, remotely controlled, and/or autonomous.

In some embodiments, sensor 140/150 may be configured to capture data as vehicle 100 travels along a trajectory. As vehicle 100 travels along the trajectory, sensor 140/150 may continuously capture data. Consistent with the present disclosure, sensor 140/150 may capture a series of data frames of a scene as vehicle 100 travels along a trajectory near or around the scene. The data frames may be transmitted to a server 160 in real-time (e.g., by streaming) , or collectively after vehicle 100 completes the entire trajectory.

Consistent with the present disclosure, sensor 140/150 may communicate with server 160. In some embodiments, server 160 may be a local physical server, a cloud server (as illustrated in FIG. 1) , a virtual server, a distributed server, or any other suitable computing device. Consistent with the present disclosure, server 160 may store an HD map.

Consistent with the present disclosure, server 160 may be also responsible for updating the HD map from time to time to reflect changes at certain portions of the map. Instead of updating the whole map using data acquired by a LiDAR and a camera, server 160 may update just the portion of the map that contains the changing object (s) using data captured as vehicle 100 travels along a trajectory near the changing object (s) . Sensor 160 may detect changed object (s) by matching the sensor data acquired by sensor 140/150 with the corresponding map data of the object (s) . Once changed object (s) are detected, server 160 may further use the acquired data of the changed object (s) to update the HD map. For example, server 160 may obtain sensor data from sensor 140/150, extract sensor data corresponding to the changed object (s) and use the extracted sensor data to update the HD map. Server 160 may communicate with sensor 140/150, and/or other components of vehicle 100 via a network, such as a Wireless Local Area Network (WLAN) , a Wide Area Network (WAN) , wireless networks such as radio waves, a cellular network, a satellite communication network, and/or a local or short-range wireless network (e.g., Bluetooth ^TM) .

For example, FIG. 2 illustrates a block diagram of an exemplary server 160 for updating an HD map, according to embodiments of the disclosure. Consistent with the present disclosure, server 160 may receive an HD map to be updated (e.g., HD map 101) from database/repository 210 and sensor data (e.g., sensor data 103) corresponding to one or more of landmarks included in HD map 101 from sensor 140/150. Based on sensor data 103, in some embodiments, server 160 may match a subset of sensor data 103 corresponding to a landmark with the map data of the landmark (referred to as the “landmark data” ) based on a neural network. Server 160 may further determine if the landmark needs to be updated based on determining a matching index indicating the difference between the subset of sensor data 103 and the corresponding landmark data associated with HD map 101. In some embodiments, if the matching index for one of the landmarks within an HD map (e.g., HD map 101) is higher than a predetermined threshold (e.g., indicating a road lane has been changed due to re-planning) , server 160 may send a survey request (e.g., request 105) to sensor 140/150 (or to vehicle 100 that carries sensor 140/150) to collect sensor data of the changed road lane. Then server 160 may update HD map 101 by reconstructing LiDAR reflection image based on a subset of sensor data 103 associated with the changed landmark. In some embodiments, server 160 may then send the updated HD map (e.g., HD map 107) back to database/repository 210 for storage.

In some embodiments, as shown in FIG. 2, server 160 may include a communication interface 202, a processor 204, a memory 206, and a storage 208. In some embodiments, server 160 may have different modules in a single device, such as an integrated circuit (IC) chip (implemented as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA) ) , or separate devices with dedicated functions. In some embodiments, one or more components of server 160 may be located in a cloud or may be alternatively in a single location (such as inside vehicle 100 or a mobile device) or distributed locations. Components of server 160 may be in an integrated device or distributed at different locations but communicate with each other through a network (not shown) .

Communication interface 202 may send data to and receive data from components such as sensor 140/150 and database/repository 210 via communication cables, a Wireless Local Area Network (WLAN) , a Wide Area Network (WAN) , wireless networks such as radio waves, a cellular network, and/or a local or short-range wireless network (e.g., Bluetooth ^TM) , or other communication methods. In some embodiments, communication interface 202 can be an integrated service digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection. As another example, communication interface 202 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented by communication interface 202. In such an implementation, communication interface 202 can send and receive electrical, electromagnetic or optical signals that carry digital data streams representing various types of information via a network.

Consistent with some embodiments, communication interface 202 may receive data such as sensor data 103 captured by sensor 140/150 and HD map 101 to be updated from database/repository 210. Communication interface may further provide the received data to storage 208 for storage or to processor 204 for processing. In some embodiments, communication interface 202 may send a survey request to any local part of vehicle 100 (or sensor 140/150 directly) and communication interface 202 may also receive an updated HD map generated by processor 204 and provide updated HD map 107 to database/repository 210 or any remote device via a network.

Processor 204 may include any appropriate type of general-purpose or special-purpose microprocessor, digital signal processor, or microcontroller. Processor 204 may be configured as a separate processor module dedicated to updating the HD map. Alternatively, processor 204 may be configured as a shared processor module for performing other functions unrelated to HD map updating.

As shown in FIG. 2, processor 204 may include multiple modules, such as an HD map segmentation unit 240, a sensor data matching unit 242, a matching index determination unit 244, and an HD map update unit 246, and the like. These modules (and any corresponding sub-modules or sub-units) can be hardware units (e.g., portions of an integrated circuit) of processor 204 designed for use with other components or software units implemented by processor 204 through executing at least part of a program. The program may be stored on a computer-readable medium, and when executed by processor 204, it may perform one or more functions. Although FIG. 2 shows units 240-246 all within one processor 204, it is contemplated that these units may be distributed among multiple processors located near or remotely with each other.

HD map segmentation unit 240 may be configured to segment HD map 101 after communication interface 202 receives HD map 101 from database/repository 210. In some embodiments, HD map 101 may be segmented into segments of the same size. For example, HD map segmentation unit 240 may segment HD map 101 into segments each of which corresponds to a 200＊200 m ² (square meters) area in real-world. After segmenting HD map 101, communication interface 202 may receive sensor data 103 including sensor data of at least one landmark within a segment acquired from sensor 140/150. Other units of sever 160 (described in detail below) may then determine a matching index of the segmentation based on the segmentation and sensor data 103, and may update HD map 101 by updating just the map segment (s) that include the changed landmarks when the matching index of at least one landmark within each segment is higher than a predetermined threshold) .

Sensor data matching unit 242 may be configured to match sensor data 103 with corresponding landmark data associated with HD map 101. In some embodiments, sensor data matching unit 242 may use a neural network to extract a subset of sensor data 103 corresponding to a landmark within HD map 101. For example, sensor data matching unit 242 may use a principal component analysis (PCA) to transform the original sensor data into a set of data representations in a lower dimensional space. Sensor data matching unit 242 may also train a support vector machine (SVM) using an MLP-kernel to classify the sensor data according to the objects contained therein. The trained SVM may be applied by sensor data matching unit 242 to extract the subset of sensor data 103 corresponding to the landmark.

Matching index determination unit 244 may determine a matching index indicating a difference between the matched subset of sensor data and the corresponding landmark data and determine if the matching index is higher than a predetermined threshold. In some embodiments, if the landmark is a ground mark (e.g., road lane marks) , matching index determination unit 244 may determine the matching index by projecting the corresponding the landmark data within HD map 101 to the subset of sensor data 103, and then determine the matching index based on the projection of the corresponding landmark data and the subset of sensor data 103. For example, matching index determination unit 244 may project road lane mark data in HD map 101 to the subset of sensor data 103 corresponding to the road lane marks. The projected data may be denoted with L ₁, and the subset of sensor data 103 corresponding to the road lane marks may be denoted with L ₂. A matching index may be determined based on calculating a slope difference between L ₁ and L ₂ and calculating a length difference between L ₁ and L ₂ (e.g., a difference in distances calculated between two ends of the landmark in the projected data L ₁ and the subset of sensor data 103 L ₂. For example, the matching index may be calculated according to equation (1) :

Score (lane) = λ (Slop) * ||Slop (L1) –Slop (L2) || + ∑ _{p ∈endpoints} Distance (p (L1) , p (L2) ) (1)

In some embodiments, if the landmark is a standing sign (e.g., traffic sign) , matching index determination unit 244 may determine the matching index by determining position information of the landmark based on a visual geometry method. For example, using the visual geometry method, matching index determination unit 244 may determine an estimated coordinate of the landmark based on the position information. Matching index determination unit 244 may then identify an object within HD map 101 corresponding to the landmark based on the estimated coordinate. The matching index may be determined based on the sensor data of the landmark and the landmark data of the identified object within the HD map. In some embodiments, matching index determination unit 244 may further determine if the type of the object matches the type of the landmark and calculate a penalty score if the types do not match. For example, matching index determination unit 244 may extract a traffic sign A and categorize the traffic sign A as Sign (A) . Matching index determination unit 244 may then use a visual geometry method (e.g., visual geometry group) to determine position information of Sign (A) . Then matching index determination unit 244 may assume a central point of Sign (A) and accordingly estimate a coordinate of the central point as P (A) . The variance of the estimated coordinate may be descried by W (A) (e.g., a 3＊3 matrix) . Matching index determination unit 244 may also identify an object B within HD map 101 corresponding to traffic sign A based on the estimated coordinate and categorize the object as Sign (B) . The coordinate of the central point may be accordingly estimated as P (B) . Then matching index determination unit 244 may determine if the type of the object matches the type of the landmark, e.g., by comparing Sign (A) with Sign (B) . If they match, matching index determination unit 244 may calculate a matching index according to equation (2) :

Score (sign) = (P (A) –P (B) ) ^T ^＊ W (A) ^＊ (P (A) –P (B) ) (2)

If they do not match, matching index determination unit 244 may assign a penalty score as the matching index according to equation (3) :

Score (sign) = Ω (sign_diff) (3)

where Ω (sign_diff) is the penalty score (e.g., may take a value within the range based on past experiences, such as 0～30) .

In some other embodiments, if the landmark is an edge of a road (e.g., a curb) , matching index determination unit 244 may determine the matching index by matching sensor data 103 with HD map data of HD map 101 based on position information, generating a 3D mesh for sensor data 103 associated with the landmark, determining a first height of each cell in the 3D mesh and determining the matching index based on the first height of each cell and a second height determined using the corresponding HD map data.

For example, matching index determination unit 244 may use GPS data associated with sensor data 103 to match sensor data 103 and map data of HD map 101. Matching index determination unit 244 may generate a 3D mesh for sensor data associated with the curb (e.g., mesh LiDAR data within the road area into 3D meshes with a 0.1 m (meter) resolution) and determine height of the meshes based on plane fitting the meshes. The height information derived from sensor data 103 may be defined as E (A) and the height information derived from the map data of HD map 101 may be defined as E (B) . Matching index determination unit 244 may determine the matching index according to equation (4) :

Score (elevation) = ∑‖E (A) -E (B) ‖ (4)

HD map updating unit 246 may be configured to determine if HD map 101 needs to be updated. In some embodiments, HD map updating unit 246 may compare the determined matching index to a predetermined threshold and may update the HD map by updating a portion of HD map 101 using the subset of sensor data 103 corresponding to the changed landmark. For example, HD map updating unit 246 may construct a LiDAR data reflection image based on sensor data 103 and extract a subset of sensor data associated with the landmark based on the LiDAR data reflection image. HD map updating unit 246 may update the HD map data of the changed landmark in HD map 101 using the corresponding extracted subset of sensor data 103.

Memory 206 and storage 208 may include any appropriate type of mass storage provided to store any type of information that processor 204 may need to operate. Memory 206 and storage 208 may be a volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other type of storage device or tangible (i.e., non-transitory) computer-readable medium including, but not limited to, a ROM, a flash memory, a dynamic RAM, and a static RAM. Memory 206 and/or storage 208 may be configured to store one or more computer programs that may be executed by processor 204 to perform HD map updating disclosed herein. For example, memory 206 and/or storage 208 may be configured to store program (s) that may be executed by processor 204 to update an HD map based on matching sensor data with the map data using a neural network.

Memory 206 and/or storage 208 may be further configured to store information and data used by processor 204. For instance, memory 206 and/or storage 208 may be configured to store the various types of data (e.g., image data, LiDAR data, etc. ) captured by sensors 140/150 and HD map 101. Memory 206 and/or storage 208 may also store intermediate data such as neural network models, map segments, and subsets of sensor data, etc. The various types of data may be stored permanently, removed periodically, or disregarded immediately after each frame of data is processed.

Updating HD map 101 when significant changes had occurred to at least one of the landmarks within HD map 101 (e.g., the matching index of at least one of the landmarks is higher than the predetermined threshold) by updating only sensor data of the changed landmark, the disclosed systems and methods may be more efficient (e.g., save more computation power and time) comparing to updating the entire HD map whenever there is a change of a landmark within the HD map. For example, the data of the unchanged objects within the HD map may stay the same and may not need to be processed again.

FIG. 3 illustrates a flowchart of an exemplary method 300 for updating an HD map, according to embodiments of the disclosure. In some embodiments, method 300 may be implemented by a map update system that includes, among other things, sensors 140/150, server 160 and database/repository 210. However, method 300 is not limited to that exemplary embodiment. Method 300 may include steps S302-S314 as described below. It is to be appreciated that some of the steps may be optional to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown in FIG. 3.

In step S302 and S304, HD map 101 may be received from database/repository 210 and may be segmented into segments of the same size. For example, server 160 may segment HD map 101 into segments each of which corresponds to a 200＊200 square meters area in real-world.

In step S306, sensor data 103 of landmarks within one segment acquired by at least one of sensor 140/150 may be received from sensor 140/150. In some embodiments, the landmark may be a ground mark (e.g., road lane marks) , a standing sign (e.g., traffic signs) or an edge of a road (e.g., curbs) .

In step S308, server 160 may match sensor data 103 with corresponding landmark data associated with the HD map. In some embodiments, server 160 may use a neural network to extract a subset of sensor data 103 corresponding to the landmark from sensor data 103. For example, server 160 may use a principal component analysis (PCA) to transform the original sensor data into a set of data representations in a lower dimensional space and may use a trained SVM to classify and to extract the subset of the sensor data corresponding to the landmark.

In step S310, server 160 may determine a matching index indicating a difference between the matched subset of sensor data and the corresponding landmark data. In some embodiments, if the landmark is a ground mark (e.g., road lane marks) , server 160 may determine the matching index by comparing features of the subset of sensor data 103 corresponding to the road lane mark and those of the landmark data of HD map 101. For example, server 160 may project road lane marks data in HD map 101 to the subset of sensor data 103 corresponding to the road lane mark (e.g., the projected data may be denoted with L ₁, and the subset of sensor data 103 corresponding to the road lane mark may be denoted with L ₂) . Server 160 may calculate a slope difference between L ₁ and L ₂ and a length difference between L ₁ and L ₂ (e.g., a difference in distances calculated between two ends of the landmark in the projected data L ₁ and the subset of sensor data 103 L ₂) to determine the matching index. For example, the matching index may be calculated according to equation (1) .

In some embodiments, if the landmark is a standing sign (e.g., traffic signs) , server 160 may determine the matching index by first comparing the types of the landmarks within sensor data 103 and HD map 101. If they match, server 160 may further calculate the matching index based on determining a coordinate estimation of a central point of the landmark within sensor data 103 and the corresponding landmark within HD map 101 based on the coordinate estimation.

For example, server 160 may extract a traffic sign A and categorize the traffic sign A as Sign (A) . Server 160 may then use a visual geometry method to determine position information of Sign (A) . Then server 160 may assume a central point of Sign (A) and accordingly estimate a coordinate of the central point as P (A) the variance of which may be described as W (A) (a 3＊3 matrix) . Server 160 may also identify an object B within HD map 101 corresponding to traffic sign A based on the estimated coordinate where the object may be categorized as Sign (B) . The coordinate of the central point may be accordingly estimated as P (B) . Then server 160 may determine if the type of the object matches the type of the landmark, e.g., by comparing Sign (A) with Sign (B) . If they match, server 160 may calculate a matching index according to equation (2) and if they do not match, server 160 may assign a penalty score as the matching index according to equation (3) where Ω (sign_diff) is the penalty score (e.g., may take a value within the range based on past experiences, such as 0～30) .

In some other embodiments, if the landmark is an edge of a road (e.g., a curb) , server 160 may determine the matching index comparing the height information of the subset of sensor data 103 corresponding to the curb and the map data of the curb within HD map 101.

For example, server 160 may use GPS data associated with sensor data 103 to match sensor data 103 and map data of HD map 101. Server 160 may generate a 3D mesh for sensor data associated with the curb (e.g., mesh LiDAR data within the road area into 3D meshes with a 0.1 m (meter) resolution) and determine height of the meshes based on plane fitting the meshes. The height information derived from sensor data 103 may be defined as E (A) and the height information derived from the map data of HD map 101 may be defined as E (B) . Server 160 may determine matching index according to equation (4) .

In step S312, the matching index may be compared to a predetermined threshold. In some embodiments, if the matching index is lower than the threshold (step S312: No) indicating that there is no significant change in the segment, method 300 may return to step S306, where sensor data 103 of landmarks within another segment may be received by server 160.

If the matching index is higher than the threshold (step S312: Yes) , in step S314, server 160 may update HD map 101 by updating the segment (s) containing the changed landmark with the subset of sensor data 103 corresponding to the changed landmark. For example, server 160 may construct a LiDAR data reflection image based on sensor data 103 and extract a subset of sensor data 103 associated with the landmark based on the LiDAR data reflection image. Server 160 may update the HD map data of the changed landmark in HD map 101 using the corresponding extracted subset of sensor data 103.

Updating an HD map by only updating the sensor data of the changed landmark may save significant computation power and time because the data of the unchanged parts of the HD map are not processed.

Another aspect of the disclosure is directed to a non-transitory computer-readable medium storing instructions which, when executed, cause one or more processors to perform the methods, as discussed above. The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of computer-readable medium or computer-readable storage devices. For example, the computer-readable medium may be the storage device or the memory module having the computer instructions stored thereon, as disclosed. In some embodiments, the computer-readable medium may be a disc or a flash drive having the computer instructions stored thereon.

It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed system and related methods. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed system and related methods.

It is intended that the specification and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.

Claims

A method for updating an HD map comprising:

receiving, by a communication interface, the HD map;

segmenting, by at least one processor, the HD map into a plurality of segments;

receiving, by the communication interface, sensor data of at least one landmark within a segment acquired by at least one sensor;

matching, by the at least one processor, the sensor data with corresponding landmark data associated with the HD map;

determining, by the at least one processor, that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold; and

updating, by the at least one processor, the segment of the HD map by updating the corresponding landmark data.
The method of claim 1, wherein the at least one sensor comprises a camera or a LiDAR.
The method of claim 1, wherein determining the matching index further comprises extracting a subset of the sensor data corresponding to the landmark based on a neural network.
The method of claim 3, wherein the landmark is a ground mark, and determining the matching index further comprises:

projecting the corresponding landmark data associated with the HD map to the subset of the sensor data; and

determining the matching index based on the projection of the corresponding landmark data and the subset of the sensor data of the landmark.
The method of claim 4, wherein the matching index is determined based on calculating a slope difference between the projection and the subset of the sensor data and calculating a distance between two ends of the landmark difference between the projection and the subset of the sensor data.
The method of claim 3, wherein the landmark is a standing sign, and determining the matching index further comprises:

determining position information of the landmark based on a visual geometry method;

determining an estimated coordinate of the landmark based on the position information;

identifying an object within the HD map corresponding to the landmark based on the estimated coordinate; and

determining the matching index based on the sensor data of the landmark and the landmark data of the identified object within the HD map.
The method of claim 6, wherein identifying an object within the HD map corresponding to the landmark further comprises:

determining if the type of the object matches the type of the landmark; and

assigning a penalty score as the matching index if the types do not match.
The method of claim 6, wherein the matching index is determined based on calculating a variance of the estimated coordinate of the landmark.
The method of claim 1, wherein the landmark is an edge of a road, and determining the matching index further comprises:

matching the sensor data with the HD map data based on position information;

generating a 3D mesh for the sensor data associated with the landmark;

determining a first height of each cell in the 3D mesh; and

determining the matching index based on the first height of each cell and a second height determined using the corresponding HD map data.
The method of claim 9, wherein the first height of each cell is determined based on a plane fit method.
The method of claim 1, wherein the landmark is at least one of a ground mark, a standing sign or an edge of a road.
The method of claim 1, wherein updating the corresponding landmark data further comprises initiating a survey trip for collecting sensor data associated with the landmark.
The method of claim 12, wherein updating the corresponding landmark data further comprises:

constructing a LiDAR data reflection image based on the sensor data; and

extracting a subset of sensor data associated with the landmark based on the LiDAR data reflection image.
A system for updating an HD map, comprising:

a communication interface configured to receive the HD map and sensor data of at least one landmark within a segment acquired by at least one sensor;

a storage configured to store the HD map and the sensor data; and

at least one processor, configured to:

segment the HD map into a plurality of segments;

match the sensor data with corresponding landmark data associated with the HD map;

determine that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold; and

update the segment of the HD map by updating the corresponding landmark data.
The system of claim 14, wherein the at least one sensor comprises a camera or a LiDAR.
The system of claim 14, wherein to determine the matching index, the at least one processor is further configured to extract a subset of the sensor data corresponding to the landmark based on a neural network.
The system of claim 16, wherein the landmark is a ground mark, to determine the matching index, the at least one processor is further configured to:

project the corresponding landmark data associated with the HD map to the subset of the sensor data; and

determine the matching index based on the projection of the corresponding landmark data and the subset of the sensor data of the landmark.
The system of claim 16, wherein the landmark is a standing sign, to determine the matching index, the at least one processor is further configured to:

determine position information of the landmark based on a visual geometry method;

determine an estimated coordinate of the landmark based on the position information;

identify an object within the HD map corresponding to the landmark based on the estimated coordinate; and

determine the matching index based on the sensor data of the landmark and the landmark data of the identified object within the HD map.
The system of claim 14, wherein the landmark is an edge of a road, to determine the matching index, the at least one processor is further configured to:

match the sensor data with the HD map data based on position information;

generate a 3D mesh for the sensor data associated with the landmark;

determine a first height of each cell in the 3D mesh; and

determine the matching index based on the first height of each cell and a second height determined using the corresponding HD map data.
A non-transitory computer-readable medium having instructions stored thereon that, when execute by one or more processors, cause the one or more processors to perform a method for updating an HD map, the method comprising:

receiving the HD map;

segmenting the HD map into a plurality of segments;

receiving sensor data of at least one landmark within a segment acquired by at least one sensor;

matching the sensor data with corresponding landmark data associated with the HD map;

determining that a matching index indicative of a difference between the sensor data and the corresponding landmark data is higher than a threshold; and

updating the segment of the HD map by updating the corresponding landmark data.