US20230111722A1

US20230111722A1 - Curb detection by analysis of reflection images

Info

Publication number: US20230111722A1
Application number: US18/048,011
Authority: US
Inventors: Yujie WEI; Xue Mei
Original assignee: Tusimple Inc
Current assignee: Tusimple Inc
Priority date: 2018-05-02
Filing date: 2022-10-19
Publication date: 2023-04-13
Also published as: US11500101B2; CN116129376A; CN110458854A; CN110458854B; US20190340447A1

Abstract

Disclosed are devices, systems and methods for road feature detection using light detection and ranging (LiDAR) sensors. One example of a method for road feature detection includes obtaining a point-cloud frame that comprises a description of an intensity of a reflection of beams from an area around a vehicle. The example method further includes creating a plurality of clusters that each include (i) one or more seed points of the point-cloud frame, and (ii) additional points of the point-cloud frame based on a relationship between the additional points and the one or more seed points. The example method further includes identifying a cluster from the plurality of clusters based on a total number of points included in each of the plurality of clusters. The example method further includes detecting a road feature from the cluster.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent document is a continuation of U.S. patent application Ser. No. 16/401,772, entitled “CURB DETECTION BY ANALYSIS OF REFLECTION IMAGES”, filed on May 2, 2019, which claims priority to and benefits of U.S. Provisional Patent Application No. 62/665,994, entitled “CURB DETECTION BY ANALYSIS OF REFLECTION IMAGES,” filed on May 2, 2018. The entire contents of the above-referenced patent applications are incorporated by reference as part of the disclosure of this patent document.

TECHNICAL FIELD

This document relates to computer-based estimation of roadside curbs by image analysis.

BACKGROUND

Autonomous vehicle navigation is a technology for sensing the position and movement of a vehicle and, based on the sensing, autonomously control the vehicle to navigate towards a destination. Autonomous vehicle navigation can have important applications in transportation of people, goods and services. One of the components of autonomous driving, which ensures the safety of the vehicle and its passengers, as well as people and property in the vicinity of the vehicle, is curb detection.

SUMMARY

Disclosed are devices, systems and methods for curb detection by analyzing images of an area surrounding a vehicle. In some embodiments, light detection and ranging (LiDAR) sensors may be used to acquire the images based on reflections captured from the surrounding area. One example implementation can locate the curb position in 3D space accurately and robustly, using point-cloud data captured by a top-mounted LiDAR sensor as input, and output a predicted curve indicating the curb's position that can further be used for autonomous navigation of the vehicle.
In one aspect, the disclosed technology can be used to provide a method for curb detection using LiDAR sensors. This method includes acquiring, based on a scan of an area around a vehicle, a point-cloud frame that includes a description of an intensity of a reflection of the area around the vehicle, selecting a subset of points from the point-cloud frame, seeding each of a plurality of clusters based on the selected points, determining a criterion for each of the selected points, increasing a size of each of the plurality of clusters by including additional points from the point-cloud frame that meet the criterion for the corresponding cluster, identifying a largest cluster of the plurality of clusters upon completion of the increasing step, and detecting a curb based on a boundary of the largest cluster.
In another aspect, the above-described method is embodied in the form of processor-executable code and stored in a computer-readable program medium.
In yet another aspect, a device that is configured or operable to perform the above-described method is disclosed. The device may include a processor that is programmed to implement this method.
The above and other aspects and features of the disclosed technology are described in greater detail in the drawings, the description and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an annotated flow diagram of an example method for LiDAR-based curb detection.

FIG. 2 shows an example of convex and concave hulls of a set of points.

FIG. 3 shows a flowchart for an example method of for LiDAR-based curb detection.

FIGS. 4A and 4B show examples of LiDAR-based curb detection in the presence of another vehicle on the road.

FIG. 5 shows an example of a hardware platform that can implement some techniques described in the present document.

DETAILED DESCRIPTION

A curb usually refers to an edge where a raised sidewalk meets a street or roadway. In other words, the curb typically reveals the boundary of the drivable area of the road. An accurate and robust curb detection is useful to an autonomous driving system to ensure safe and lawful driving of a vehicle. LiDAR-based curb detection has been an active area of research and implementation in remote sensing and autonomous driving in recent years. For example, curb detection is also useful for autonomous trucks, which typically travel on highways from an origin distribution center to a destination distribution center, but necessarily travel through surface streets for the last mile at the start and end of their journey.
Conventional curb detection implementations use single or accumulated point-cloud frames as input, and employ pointwise features, such as height difference or LiDAR ring radius compression, to detect the curb. However, these conventional methods are focused on directly detecting the curbs, and are therefore susceptible to “curb-like” noises like vehicles on the road or bushes on the roadside.
LiDAR-based curb detection methods face many challenges. For example, most of the curb features (e.g. height difference, ring radius compression, and normal change) become weaker (more difficult to distinguish) when the curb is far away from the sensor. Furthermore, the methods must contend with occlusions and roadside objects having similar features to curbs, and the diversity of curb types that may require laborious parameter tuning in any conventional curb detection method.
Embodiments of the disclosed technology can locate the position of the curb in 3D space based on (a) detecting the curb using road surface expansion to find the boundary of the road instead of directly detecting the curb itself, which is advantageously robust to the noise and occlusions from the road surface, and (b) combining multiple features for robust point clustering.
FIG. 1 shows an annotated flow diagram of an example method for LiDAR-based curb detection. The flow diagram 100 includes using a LiDAR sensor (block 110) mounted on top of a vehicle, which emits laser beams, to generate point-cloud data of the surrounding environment. In an example, the subfigure 112 shows an exemplary point-cloud generated based on the LiDAR sensor. In some embodiments, the point-cloud includes (x, y, z) coordinates in a coordinate system and the intensity of the reflection of the laser beams received at the sensor.
For example, the point-cloud may be a stored as a pixel array of four dimensions, with three dimensions corresponding to the spatial coordinates of the scanned surrounding area and one dimension representing a strength of the reflection sensed by the LiDAR sensors. In some embodiments, a multi-dimensional array may be used for storage of the point-cloud data, with three spatial dimensions, and additional dimensions corresponding to multiple sensors and their corresponding reflection map.
The flow diagram 100 includes a pre-processing step that takes the single-frame point-cloud from the LiDAR sensor and motion information from the Global Positioning System (GPS) unit and the Inertial Measurement Unit (IMU) (block 105) to register consecutive point-cloud frames into a common coordinate system, which may then be accumulated (block 110). The accumulated point-cloud (e.g. shown in subfigure 122) is typically much denser than the single-frame point-cloud.
The flow diagram 100 includes the curb detection module (block 130) that estimates features for a subset of the points in the accumulated point-cloud. In some embodiments, the features include the normal at each point, which is estimated by fitting a plane to the point and its nearby points within a certain range (referred to as a “point group”). In other embodiments, the features include the curvature of the point group, which is estimated as λ₀/(λ₀+λ₁+λ₂), where λ₀<λ₁<λ₂are the eigenvalues of the covariance matrix formed by the nearby points. In some embodiments, a k-dimensional tree may be constructed from the accumulated point-cloud to support faster, and more efficient, nearest neighbor searches.
The curb detection module then randomly selects some points as seeds to create clusters, and region-growth is performed from these seeds. In the region-growing process, a point cluster will check its neighboring points to see if they satisfy a criterion associated with height, the normal, and curvature. The points that satisfy the criterion will be added to the cluster, and may become new seeds that are used to expand the cluster. A cluster will be removed from the original accumulated point-cloud when it stops growing, and the curb detection module will start a new cluster from another random seed among the remaining points. In some embodiments, the process may terminate when the total number of remaining points from the original accumulated point-cloud drops below a certain threshold, resulting, for example, in subfigure 132.
In some embodiments, satisfying the criterion may correspond to all the points in a region being connected in some predefined sense. In an example, a smoothness requirement (or condition) needs a value of a certain metric computed for each point to be within some tolerance of an initial or average value. In another example, the criterion may depend on multiple metrics with varying numerical range requirements.
In some embodiments, a 4-connected neighborhood (Von Neumann neighborhood) may be used in the region-growing step. In other embodiments, a 8-connected neighborhood (Moore neighborhood) may be used to grow the region from the initially selected seed points.
The curb detection module relies on the assumption that the largest cluster close to the driving trajectory will always be the drivable area, e.g. the road surface. This assumption holds since the origin of the point-clouds (both the single-frame and accumulated versions) is the center of the LiDAR sensor, which is co-located with the vehicle and its driving trajectory, and provides very strong a priori knowledge about where the road surface will be. Thus, the curb detection module may extract the points that belong to the drivable area by selecting the cluster that contains the most points.
The boundary of the road surface is typically the location of the curb. To obtain the boundary of the drivable area, the curb detection module may compute the concave hull of the largest cluster. FIG. 2 shows an example of a convex hull (210) and a concave hull (220) of a set of points. Using the concave hull for curb detection is more robust to outliers and noise than using the convex hull, as shown in FIG. 2 . Therein, an outlying point may significantly skew the detection of the curb (205) if a convex hull is used (210), but has a minimal impact when employing a concave hull (220) in the computations. The concave hull computation is followed by the two curves on either side of the road surface being fit using, for example, the random sample consensus (RANSAC) method to find the left and right curbs of the road.
The flow diagram 100 includes a post-processing module (block 140) that uses the curves generated from multiple frames, and fuses that information based on their temporal continuity (to generate, for example, subfigure 142). In some embodiments, the post-processing model may smooth out the curb points using a b-spline fitting function.
FIG. 3 shows a flowchart for an example method 300 for LiDAR-based curb detection. The method 300 includes, at step 310, acquiring a point-cloud frame. In some embodiments, the point-cloud frame is acquired based on a scan of an area around a vehicle, and may include a description of an intensity of a reflection of the area around the vehicle. For example, the scan of the vehicle may be performed by a LiDAR sensor that is spinning a rate of 20 Hz. Due to the use of high frequency, high velocity waves for sensing reflections, the LiDAR is typically not affected by the speed of the vehicle, and may support vehicle speeds ranging from 5 miles per hour (mph) to 70 mph. In some embodiments, the LiDAR scanning is based on a rotation rate parameter, an image resolution parameter and/or a frame rate parameter.
In some embodiments, the point-cloud frame may be generated based on the accumulation of multiple single-frame point-clouds. As expected, the accumulated point-cloud frame is much denser than any individual single-frame point-cloud. This enables more robust curb detection, and pre- and post-processing, since noise effects are averaged out due to the accumulation.
The method 300 includes, at step 320, selecting a subset of points from the point-cloud frame, and further includes, at step 330, seeding each of a plurality of clusters based on the selected points, which are the preliminary steps of the region-growing method. The selection process may be implemented in a variety of different ways. For example, in some embodiments, the selected subset may be uniformly distributed throughout the 3D space represented by the point-cloud. In some embodiments, the subset of points may be a fixed pre-determined number (e.g., 10,000 points), and more points may be selected from an area that was detected previously to include a previous estimate of the curb. Advantageously, such a selection allows more computational power to be applied to the area that has a higher probability of including the border between the road surface and the curb. In some embodiments, curb detection may be performed periodically as the vehicle moves along a road, and the subset of points may be selected to be non-co-located compared to a previous selection. Advantageously, such a selection provides a thorough and uniform coverage of the surrounding area over a number of curb detections performed by the system.
The method 300 includes, at step 340, determining a criterion for each of the selected points. In some embodiments, the criterion may include a normal vector and a curvature, which may be computed for each of the selected points and based on neighboring points. In an example, the curvature may be computed as λ₀/(λ₀+λ₁+λ₂), where λ₀<λ₁<λ₂are the eigenvalues of the covariance matrix formed by the neighboring points.
The method 300 includes, at step 350, increasing a size of each of the plurality of clusters by including additional points from the point-cloud frame that meet the criterion for the corresponding cluster. The region-growing method used to increase the size of the clusters is a segmentation method that examines neighboring points (or pixels) of initial seed points and determines whether the point neighbors should be added to the region. The process is iterated on, in the same manner as general data clustering algorithms.
In some embodiments, and in the context of FIG. 1 , the criterion may be referred to as a smoothness condition that is implemented using one or more features and corresponding thresholds or tolerances. For example, neighboring points that have feature values that are within a certain tolerance of the seed point feature values are added to that cluster.
The method 300 includes, at step 360, identifying, upon completion of the increasing step 350, a largest cluster of the plurality of clusters. Since the LiDAR sensor is co-located with the vehicle and its driving trajectory, there is very strong a priori knowledge, or a high statistical probability, that the largest cluster will correspond to the road surface, and be very close to the driving trajectory (e.g. within a foot, or within 2-20% of the width of the vehicle) or even enclose the driving trajectory. Thus, when the region (or cluster) growing operation is completed, the largest cluster is identified as the road surface. In an example, the region-growing operation may be terminated based on the number of points of the point-cloud frame that have not been considered. In another example, the stopping condition for the region-growing method may be the size of the largest cluster exceeding a certain threshold.
The method 300 includes, at step 370, detecting a curb based on a boundary of the largest cluster. The boundary of the largest cluster, which is typically the road surface, will be the curb location. In some embodiments, the method 300 includes smoothing the boundary of the largest cluster based on a spline function. In other embodiments, the method 300 further includes computing the concave hull of the largest cluster.
In some embodiments, the method 300 may further include determining whether any of the clusters include bounded regions of points, where each point in a bounded region does not meet the criterion. In other words, the region-growing method may have grown the largest cluster around another vehicle on the road, which results in a bounded region with feature values that different from the feature values of the largest (or road) cluster.
Since the bounded region includes points that do not meet the criterion, the method is able to correctly identify that these points are not representative of the curb, but rather of another object in the road (e.g. a vehicle), and the method 300 further includes either adding the points of the bounded region of points to the corresponding cluster, or correctly identifying them as not being part of the curb. This operation ensures that embodiments of the disclosed technology can accurately and robustly detect the curb.
In some embodiments, the movement of the object in the road may be tracked as the vehicle moves along the road. If curb detection is performed periodically, the tracking information may be leveraged to provide a priori information to the region-growing method, which would advantageously result in faster and more accurate processing.
FIGS. 4A and 4B show examples of LiDAR-based curb detection in the presence of another vehicle on the road. Embodiments of the disclosed technology are advantageously able to accurately detect the curb in the presence of an obstruction such as another vehicle on the road. As shown in FIG. 4A, the ego-vehicle 403 (the term “ego-vehicle” is used to refer to autonomous vehicle that is the primary subject) that includes the LiDAR sensor 410 may be adjacent to another vehicle 413, which is closer to the curb 405 than the ego-vehicle. Due to the view-point of the ego-vehicle 403, the other vehicle 413 is detected as “an L-shape” (as shown by the “×” marks in FIG. 4A) and a portion of the curb is also detected (as shown by the “*” marks in FIG. 4A).
In some embodiments, the detected points corresponding to the vehicle (the “×” marks in FIG. 4A) may be removed from consideration, and the curve fitting may be applied to the detected points corresponding to the curb (the “*” marks in FIGS. 4A and 4B) in order to accurately identify the curb, including the portion that was occluded by the vehicle 413. For example, the curve fitting used to identify the curb may be similar to what is used in the post-processing module 140 in FIG. 1 (e.g. spline interpolation).
In some embodiments, curve fitting the detected points corresponding to the curb (the “*” marks in FIG. 4A) may not be viable since the gap may be too large for a curve fitting algorithm to accurately overcome. In this scenario, some embodiments may combine the results from multiple runs of the detection algorithm. For example, over multiple runs, the area occupied by another vehicle in one run may be free in another run. Thus, the gaps in the curb may be filled in by combining the results over multiple runs. Additionally, any curb detection false positives may be eliminated by using majority voting, mean filtering and/or median filtering over results from the multiple runs.
FIG. 4B shows a scenario in which the ego vehicle 403 is positioned closer to the curb 405 than the other vehicle 413, and the LiDAR sensor 410 is able to distinguish both the other vehicle as well as the curb. In this scenario, as discussed previously, a bounded region that includes points that do not meet the criterion correspond to the other vehicle 413, and may be removed from consideration. As shown in the context of FIGS. 4A and 4B, embodiments of the disclosed technology are able to accurately and robustly detect the curb in the presence of other vehicles or occlusions on the road.
FIG. 5 shows an example of a hardware platform 500 that can be used to implement some of the techniques described in the present document. For example, the hardware platform 500 may implement the method 300 or may implement the various modules described herein. The hardware platform 500 may include a processor 502 that can execute code to implement a method. The hardware platform 500 may include a memory 504 that may be used to store processor-executable code and/or store data. The hardware platform 500 may further include a communication interface 506. For example, the communication interface 506 may implement one or more of the communication protocols (LTE, Wi-Fi, and so on) described herein.
Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.
Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

What is claimed is:

1. A method for detecting road features, comprising:

obtaining a point-cloud frame that comprises a description of an intensity of a reflection of beams from an area around a vehicle;

creating a plurality of clusters that each include (i) one or more seed points of the point-cloud frame, and (ii) additional points of the point-cloud frame based on a relationship between the additional points and the one or more seed points;

identifying a cluster from the plurality of clusters based on a total number of points included in each of the plurality of clusters; and

detecting a road feature from the cluster.

2. The method of claim 1, further comprising:

determining that the additional points are related to the one or more seed points based on determining that the additional points are neighboring to the one or more seed points and meet a criterion associated with the one or more seed points.

3. The method of claim 2, wherein the additional points meet the criterion based on having values that are within a tolerance of values of the one or more seed points.

4. The method of claim 1, wherein identifying the cluster includes:

removing certain clusters of the plurality of clusters based on a growth of a certain clusters stopping during a region-growing process in which the additional points are added to each cluster, and

identifying the cluster from remaining clusters of the plurality of clusters.

5. The method of claim 1, the additional points included in a given cluster are determined using a k-dimensional tree constructed based on features of each point of the point-cloud frame.

6. The method of claim 1, wherein detecting the road feature comprises detecting a drivable area for the vehicle from a portion of the area around the vehicle that is spanned by the cluster.

7. The method of claim 1, wherein detecting the road feature comprises detecting, according to a boundary of the cluster, a curb of a road, wherein the vehicle is located on the road.

8. The method of claim 1, wherein obtaining the point-cloud frame comprises:

acquiring two consecutive single-frame point-clouds; and

determine an accumulated point-cloud by registering the two consecutive single-frame point clouds into a common coordinate system, wherein the point-cloud frame includes the accumulated point-cloud.

9. An apparatus for detecting road features, comprising:

a processor; and

a memory comprising executable code that, when executed by the processor, causes the apparatus to:

obtain a point-cloud frame that comprises a description of an intensity of a reflection of beams from an area around a vehicle;

create a plurality of clusters that each include (i) one or more seed points of the point-cloud frame, and (ii) additional points of the point-cloud frame based on a relationship between the additional points and the one or more seed points;

identify a cluster from the plurality of clusters based on a total number of points included in each of the plurality of clusters; and

detect a road feature from the cluster.

10. The apparatus of claim 9, wherein the relationship between the additional points and the one or more seed points includes the additional points meeting a criterion associated with the one or more seed points.

11. The apparatus of claim 9, wherein a given cluster includes the additional points based on the additional points having feature values that are within a tolerance of corresponding feature values of the given cluster, wherein the feature values include a curvature value and a normal value.

12. The apparatus of claim 9, wherein identifying the cluster includes:

including, by a region-growing process, the additional points for each of the plurality of clusters,

remove a certain cluster from the plurality of clusters based on a lack of additional points to include for the certain cluster, and

identifying the cluster from remaining clusters of the plurality of clusters.

13. The apparatus of claim 9, wherein

construct a k-dimensional tree from the point-cloud frame, and

include the additional points in a given cluster based on a nearest neighbor search that uses the k-dimensional tree.

14. The apparatus of claim 9, wherein detecting the road feature comprises detecting a road surface from a portion of the area around the vehicle that is spanned by the cluster.

15. The apparatus of claim 9, wherein detecting the road feature comprises detecting, according to one or more concave hulls of the cluster, one or more curbs of a road, wherein the vehicle is located on the road.

16. The apparatus of claim 9, wherein obtaining the point-cloud frame comprises:

acquiring a plurality of consecutive single-frame point-clouds using a light detection and ranging (LiDAR) sensor;

determine the point-cloud frame to include the plurality of consecutive single-frame point-clouds using a common coordinate system.

17. A non-transitory computer-readable medium storing a program that causes a computer to execute a process, the process comprising:

detecting a road feature from the cluster.

18. The non-transitory computer-readable medium of claim 17, wherein the additional points are neighboring to the one or more seed points and meet a criterion associated with the one or more seed points.

19. The non-transitory computer-readable medium of claim 17, wherein the road feature includes a drivable area for the vehicle, wherein the drivable area is captured by a span of the cluster.

20. The non-transitory computer-readable medium of claim 17, wherein the road feature includes a curb of a road, wherein the curb is captured by a boundary of the cluster, and wherein the vehicle is located on the road.