TWI745204B - High-efficiency LiDAR object detection method based on deep learning - Google Patents

Abstract

The present invention provides a high-efficiency LiDAR object detection method based on deep learning. The steps include: (A) Obtaining a three-dimensional point cloud data from an LiDAR, the three-dimensional point cloud data is Nx4 dimensional information, and N is LiDAR The number of points, each luminous point has spatial information of x-axis value, y-axis value, z-axis value and reflection intensity information; (B) the spatial information of the three-dimensional point cloud data is rotated and displaced by the spatial conversion matrix; (C) ) Do one-dimensional convolution of the three-dimensional point cloud data multiple times to expand the feature dimension, and then pool to obtain high-dimensional features; (D) map the three-dimensional point cloud data to a two-dimensional image to extract the two-dimensional point cloud Features; and (E) input the high-dimensional features and the two-dimensional feature of the point cloud into the adjustment model for feature adjustment.

Description

High-efficiency LiDAR object detection method based on deep learning

The present invention relates to a high-efficiency lidar object detection method based on deep learning, and more particularly to a high-efficiency lidar object detection method based on deep learning that accelerates and improves object feature extraction.

Among the existing object detection methods, image-based methods have been available for years. There have been many methods, whether through traditional image recognition or deep learning. In the future, self-driving cars will use the point cloud obtained by optical radar (Lidar) to identify it as a more reliable way. Although the detection distance is not as good as millimeter wave radar, the optical radar has a higher spatial resolution enough to target objects. To distinguish objects by appearance, optical radar has become an important sensor and is widely used in autonomous driving.

In the prior art, it has been proposed to extract the artificial features of the optical radar to obtain a feature vector and classify the pedestrians in the office. In addition, in the three-dimensional deep learning method, the point cloud image has been converted into volume pixels. (Voxel), using the features calculated by the 3D Convolutional Neural Network to classify the three-dimensional point cloud data as tags. Furthermore, it is also extracted through convolution and pooling to maintain the invariance of the point cloud order Strong characteristics, Its features can be used for classification or semantic segmentation.

Most of the above technologies have achieved good results with methods based on visual images, but these methods actually achieve good results under clear image conditions, but the actual conditions are worse, including weather or environmental brightness. The situation will cause the image to become unclear, so the recognition effect will be worse, and the optical radar we will use is highly robust to some external conditions, but even though the optical radar has 50 meters However, due to the angle divergence, the point cloud data will become sparser as the detection distance becomes longer, making obstacles more difficult to identify. This is also one of the difficulties in optical radar detection.

Another point is that the processing structure of LiDAR data is very cumbersome. Compared with the pixelated structure of the image, the objects collected by LiDAR are only point clouds, which are not structured point collections. They are aimed at specific objects. Under the circumstances, it is possible to establish features related to a specific object to make a judgment, but this feature is not very accurate in other objects, and the emerging deep learning methods will solve the problem of artificial features. Recently, people have imported them into This type of 3D data is usually done through a voxelized point cloud structure, but the process of voxelization is too time-consuming. This type of method is not suitable for use in the field of self-driving cars that requires both real-time operation and accuracy.

In summary, the current voxelized network model is complicated and time-consuming. Therefore, the applicant in this case has developed a network architecture based on deep learning and a high-efficiency lidar object detection method through painstaking research. Directly to 3D The point data is processed to obtain a concise and fast three-dimensional feature, which we call Fast LiDar Net (Fast LiDar Net).

In view of the shortcomings of the above-mentioned known technology, the main purpose of the present invention is to provide a high-efficiency LiDAR object detection method based on deep learning. By proposing a network architecture, by directly processing three-dimensional point data, a simple and fast method is obtained. The three-dimensional features of the voxel solve the shortcomings of the current voxelized network model, which is both complicated and time-consuming.

In order to achieve the above objective, according to a solution proposed by the present invention, a method for detecting objects with high efficiency based on deep learning is provided. The steps include: (A) Obtaining three-dimensional point cloud data from a LiDAR, the three-dimensional point cloud data Nx4 dimensional information, N is the number of luminous points, each luminous point has spatial information of x-axis value, y-axis value, z-axis value and reflection intensity information; (B) the three-dimensional point is rotated and displaced by the space conversion matrix Spatial information of cloud data; (C) Do one-dimensional convolution of three-dimensional point cloud data multiple times to expand the feature dimension, and then pool to obtain a high-dimensional feature; (D) map the three-dimensional point cloud data into two dimensions The two-dimensional feature of the point cloud is extracted from the image; and (E) the high-dimensional feature and the two-dimensional feature of the point cloud are input into an adjustment model for feature adjustment.

Preferably, in step (D), different expansion coefficients can be used for the two-dimensional image according to the distance expansion.

Preferably, in this step (D), one of the two-dimensional images can be shadowed The image three channels give meaningful values of reflection intensity information and reflection point distance.

Preferably, the three channels of the image may include: a blue channel, a green channel, and a red channel. The blue channel is the reflection intensity of the light reaching point and adjusted to 0~255 according to the value range. If it is the light reaching projection In the image outline, fill the green channel with a value of 255, and the red channel is the distance between the luminous point and the center of the two-dimensional image, and adjust it to 0~255 according to the value range.

Preferably, the method can be integrated with a robot operating system and installed in an embedded system.

Preferably, in step (A), the ground removal step can be further performed, using Random Sample Consensus (RANSAC) to remove the ground of the LiDAR, and before using the random sample consensus, through the point cloud Voxel Filter downsampling makes the ground have the same y-axis value.

Preferably, an object grouping step can be further performed to group according to the distance between the luminous points, and search through KD Tree. When the distance between the luminous points is less than 0.2m, the two luminous points are marked as the same group .

The above summary, the following detailed description and the accompanying drawings are all intended to further illustrate the methods, means and effects adopted by the present invention to achieve the intended purpose. The other objectives and advantages of the present invention will be described in the following description and drawings.

S1-S5, S11-S15: steps

The first figure is a flowchart of object feature extraction in a high-efficiency lidar object detection method based on deep learning of the present invention.

The second figure is a flowchart of a high-efficiency lidar object detection method based on deep learning of the present invention.

The third figure is a comparison schematic diagram of the present invention after adding the reflection intensity information of the light reaching point and the distance of the reflection point.

The fourth figure is a schematic diagram of the Fast LiDar Net architecture of the present invention.

The following is a specific example to illustrate the implementation of the present invention. Those familiar with the art can easily understand the advantages and effects of the creation from the content disclosed in this specification.

Please refer to the first figure for the object feature extraction flowchart of a high-efficiency lidar object detection method based on deep learning of the present invention, and the second figure is a high-efficiency lidar object detection method based on deep learning of the present invention Flow chart of the measurement method. The present invention is to provide a high-efficiency LiDAR object detection method based on deep learning, including: step S1, obtaining three-dimensional point cloud data from a LiDAR (that is, the data collection in step S11), and the three-dimensional point cloud data is Nx4-dimensional Information, N is the number of luminous points, each luminous point has spatial information of x-axis value, y-axis value, z-axis value, and reflection intensity information. The detailed steps of object feature extraction in step S14 include: step S2, the spatial information of the 3D point cloud data is rotated and displaced by the space conversion matrix. In more detail, in order to speed up the overall model, we first do in the aspect of 3D deep learning features Change, compared to the previous voxelized The network model is complicated and time-consuming. We propose a network architecture to directly process the three-dimensional point data to obtain a simple and fast three-dimensional feature. Firstly, only the xyz axis of Lidar’s spatial information is estimated through Transform Net. Spatial transformation matrix. After preprocessing the three-dimensional point cloud data through this spatial transformation matrix, all point cloud objects of the same kind should be more consistent in space. Step S3, the three-dimensional point cloud data is subjected to multiple one-dimensional convolutions to expand the feature dimension, and a high-dimensional feature is obtained by pooling. This feature is conducive to quickly acquiring three-dimensional information. The structure from step S2 to step S3 As shown in the fourth figure, we call it Fast LiDar Net. We use the improved Fast lidar net deep learning network for feature extraction, replacing the previous deep learning with very high computational complexity. Features and artificial features effectively improve computing performance. Step S4: Map the 3D point cloud data to the 2D image to extract the 2D features of the point cloud. Although the fast LiDAR network architecture is helpful to understand the 3D spatial information, the accuracy rate is not so high, which will reduce the recognition rate. Therefore, the two-dimensional image of step S4 is integrated to improve our accuracy. This is more helpful in terms of accuracy than pure three-dimensional information, and the speed will not be reduced too much. It can be regarded as a speed and accuracy compatible approach. And step S5, input high-dimensional features and two-dimensional point cloud features into the adjustment model for feature adjustment. In detail, this step inputs the calculation time of each feature and the uniqueness and effectiveness of the feature into this model for feature adjustment. The model selects the efficient feature value fusion, and can obtain fast and accurate results. In order to achieve real-time optical radar object detection, the feature fusion method of this method separates the 20 most likely detection objects from a single point cloud. Objects and classification can reach about 80 millimeters The processing speed in seconds, converted to 14FPS, and has an accuracy rate of 94.3%.

Above, in the object feature extraction in step S14, in addition to extracting features from the 3D point information of LiDAR, we also map a 2D image to the 3D point cloud in this step S14 to extract the 2D features of the point cloud.

In this embodiment, in step S4, different expansion coefficients can be used for the two-dimensional image according to the distance expansion, and reflection intensity information and meaningful values of the reflection point distance can be assigned to one of the three channels of the two-dimensional image. In more detail, especially in the process of Lidar's conversion of two-dimensional images, due to Lidar’s mechanical characteristics, the sparseness of point clouds at different distances is different. In order to make Lidar images suitable for deep learning image recognition, At different positions of the object, we use different expansion coefficients to solve the problem of two-dimensional image holes. Specifically, we double the expansion coefficient every five meters, and finally find the outline of the object and fill it up, and we also Assign meaningful values to the three channels of the image, and add the original single-channel image to the optical radar’s reflection intensity information and the physical meaning of the reflection point distance to turn it into a three-channel image; here you can imagine that the original single-channel image is gray The first-level image is now changed to a three-channel color image (RGB), but instead of RGB color information in the three channels, the reflection intensity information of the optical radar and the reflection point distance have physical meaning. The blue channel is Lidar The reflection intensity of the point is adjusted to 0~255 according to the value range; if it is within the outline of the projection image, a value of 255 is filled in the green channel; the red channel is the distance between the light reach point and the center, and is adjusted to 0~ according to the value range. 255. The value of these three channels can help our deep learning model to identify objects more effectively (as shown in the third figure). Show).

In this embodiment, the method of the present invention can be integrated with a robot operating system and installed in an embedded system. This method does not need to consume a large amount of computer computing resources, and can be executed on a simple embedded device to achieve good execution efficiency.

In this embodiment, the ground removal step S12 can be further performed, using Random Sample Consensus (RANSAC) to remove the LiDAR ground, and before using the random sample consensus, through point cloud voxelization (Voxel Filter) Downsampling makes the ground have the same y-axis value. In more detail, the ground removal in step S12 is for effective grouping in the next step S13. The ground ring of Lidar needs to be segmented first, and the non-ground ring part is passed to the next step for grouping. Here we use random Random Sample Consensus (RANSAC) is used to remove the ground, and before using RANSAC, the Voxel Filter downsampling makes the ground have the same y-axis value as much as possible, which can also speed up and make RANSAC segment the ground more accurately.

In this embodiment, an object grouping step S13 can be further performed, which is grouped according to the distance between the light arrival points, and the KD Tree is used as a search method. When the distance between the light arrival points is less than 0.2m, the two light arrival points are marked For the same group, the length, width, and height of each group are finally screened to match the correct object size.

In this embodiment, an object classification step S15 can be further performed, and finally the features are captured and integrated through the fully connected layer, and further Classification of one-hot vector. Using the features extracted after separating each object for deep learning classification, the classification and location of objects around the car body can be established.

In summary, the present invention uses a fast three-dimensional depth model to obtain the advantages of features in three-dimensional space, while the LiDAR projection image uses the deep learning basis of the previous two-dimensional images to retain the two-dimensional image object identification. The advantage is to combine the two-dimensional and three-dimensional manually set features to adjust the appropriate execution efficiency through a model that adjusts the accuracy and execution efficiency. The adjustment model organizes all the features and then extracts these features according to the influence of speed and accuracy. The more advantageous features allow the entire classifier to have fast and accurate results after adjustment. In addition, the optical radar has high directivity and does not need to consider the influence of the light environment. It emits high-precision laser beams to scan the surroundings of the road environment. In recent years, automatic driving has been widely discussed. Relying on the development of optical radar and the advancement of hardware equipment, we can have better environmental resistance than pure image recognition. Relying on our algorithm, we can further enhance the detection speed and accuracy of surrounding objects in the environment. Moreover, in the future self-driving car system, equipped with optical radar will enable the computer system to have the ability to accurately judge surrounding objects regardless of day or night, and the optical radar has a detection capability of at least 50 meters and a 360-degree range. Compared with image recognition, it has better dynamic discrimination ability and is very suitable for the use of self-driving cars. The present invention proposes optical radar point cloud object detection based on deep learning in order to provide more comprehensive self-driving car applications. Vision perception ability.

The above-mentioned embodiments are only illustrative to illustrate the characteristics of this creation and The effect is not intended to limit the scope of the essential technical content of the present invention. Anyone familiar with this technique can modify and change the above-mentioned embodiments without violating the spirit and scope of creation. Therefore, the scope of protection of the rights of the present invention should be listed in the scope of patent application described later.

S1-S5: steps

Claims (7)

A high-efficiency LiDAR object detection method based on deep learning. The steps include: (A) Obtaining a three-dimensional point cloud data from an LiDAR, the three-dimensional point cloud data is Nx4 dimensional information, and N is the number of LiDAR points, Each LiDAR point has spatial information of x-axis value, y-axis value, z-axis value, and reflection intensity information; (B) The three-dimensional point cloud data is processed by a fast LiDAR network architecture to obtain a high Dimensional features; (C) mapping the three-dimensional point cloud data to a two-dimensional image to extract two-dimensional features of one point cloud; and (D) inputting the high-dimensional features and the two-dimensional features of the point cloud into an adjustment model as features Adjustment, wherein the steps of the fast LiDAR network architecture include: (B1) rotationally displacing the spatial information of the three-dimensional point cloud data by a spatial transformation matrix; and (B2) performing one of the three-dimensional point cloud data multiple times The dimensional convolution expands the feature dimensionality, and the high-dimensional feature is obtained by pooling. As described in item 1 of the scope of the patent application, a high-efficiency detection method for an optical object based on deep learning, wherein in the step (C), different expansion coefficients are used for the two-dimensional image according to the distance expansion. As described in item 1 of the scope of patent application, a high-efficiency LiDAR object detection method based on deep learning, wherein in step (C), the two-dimensional image One image and three channels give meaningful values to the reflection intensity information and a reflection point distance. As described in item 3 of the scope of patent application, a high-efficiency LiDAR object detection method based on deep learning, wherein the three channels of the image include: a blue channel, a green channel and a red channel, and the blue channel is the The reflection intensity of the luminous point is adjusted to 0~255 according to the value range. If it is within the contour of the luminous projection image, fill in the green channel with a value of 255, and the red channel is the difference between the luminous point and the two-dimensional image The center distance is adjusted to 0~255 according to the value range. A high-efficiency lidar object detection method based on deep learning as described in item 1 of the scope of patent application, wherein the method adopts a robot operating system to integrate and set up an embedded system. A high-efficiency lidar object detection method based on deep learning as described in item 1 of the scope of patent application, wherein in this step (A), a ground removal step is further performed, using a random sample consensus (Random Sample Consensus, Referred to as RANSAC) to remove the LiDAR ground, and before using the random sample consensus, the ground has the same y-axis value through the point cloud voxelization (Voxel Filter) down-sampling. As described in item 6 of the scope of patent application, a high-efficiency LiDAR object detection method based on deep learning is further performed with an object grouping step, which is grouped according to the distance between the optical reach points, and the search method is performed through KD Tree. when When the distance between the light reaching points is less than 0.2m, the two light reaching points are marked as the same group.

Images