CN113219475A - Method and system for correcting monocular distance measurement by using single line laser radar - Google Patents

Abstract

The present invention provides a method and system for correcting monocular ranging by using a single-line laser radar. The method includes: acquiring an image collected by the monocular camera; inputting the image into a pre-trained ranging neural network model to obtain the The distance information of the target in the image; the ranging neural network model is obtained by training the training data set and the global reference error matrix determined by the multiple lidar ranging, and the training data set includes multiple images and each the actual distance of the pixels in the image. The invention adopts the combination of laser radar and monocular camera, and uses multiple laser radars to provide multi-point accurate distance information to correct the global error of monocular ranging, which can simplify the algorithm difficulty of ranging, and at the same time improve the accuracy and reduce the cost. And the integration with various monocular ranging algorithms is good.

Description

Method and system for correcting monocular ranging by using single-line lidar

technical field

The present invention relates to the technical field of ranging, and in particular, to a method and system for correcting monocular ranging by using a single-line laser radar.

Background technique

At present, the commonly used ranging solution for unmanned vehicles is to map the target from a two-dimensional image to a three-dimensional point cloud, so as to draw the position and boundary of the target in three-dimensional space, that is, the depth information is mainly obtained by the continuously rotating lidar. The speed and wiring harness put forward higher requirements. Although the accuracy of lidar is better, the cost is also greatly increased. If only monocular ranging is used, the accuracy of existing monocular ranging is insufficient.

SUMMARY OF THE INVENTION

The invention solves the problem that the ranging scheme cannot take into account the accuracy and the cost.

In order to solve the above problems, the present invention provides a method for correcting monocular ranging by using a single-line lidar, which is applied to a ranging system including a monocular camera and a plurality of lidars, the monocular camera and the plurality of lidars. Arranged at a preset position, the method includes: acquiring an image collected by the monocular camera; inputting the image into a pre-trained ranging neural network model to obtain distance information of the target in the image; The network model is obtained by training a training data set and a global reference error matrix determined by the plurality of lidar ranging, and the training data set includes a plurality of images and the actual distances of pixels in each of the images.

Optionally, the training process of the ranging neural network model is as follows: extracting pixel features of each image in the training data set; coupling the pixel feature corresponding matrix and the global reference error matrix of the image and then inputting the ranging. The neural network model is trained until the loss function satisfies the preset termination condition.

Optionally, the optical axis of the monocular camera is arranged parallel to the ray direction of each of the lidars; at least one of the lidars is arranged close to the monocular camera, and the rest of the lidars are evenly arranged around the monocular camera. layout.

Optionally, the coupling formula of the pixel feature corresponding matrix and the global reference error matrix of the image is as follows:

Wherein, L _n is the distance error matrix after coupling, γ is the correction coefficient, L ₀ is the global reference error matrix determined by the multiple lidar ranging, and S _n is the distance between all pixels in the image and the center point of the image. matrix, O _n is the matrix composed of the ratio of any fuzzy circle in the image to the fuzzy circle at the center of the image or the fuzzy circle close to the center of the image, k 1 and k 2 are the pre-calibrated times, and ⨀ represents the element-wise product operation.

Optionally, the error between the ranging result of the lidar close to the monocular camera and the ranging result of the monocular camera is used as the global reference error L ₀ ′ , and the uniformly arranged lidar around the monocular camera is used. The error between the ranging result and the monocular camera ranging result is taken as the local reference error L ₁ , L ₀ is as follows:

Among them, λ is the influence coefficient of global reference error and local reference error.

Optionally, the method further includes: performing polynomial fitting on errors at different positions measured along a straight line in the radial direction to determine the relative position order k 1; The error is multi-form fitting to determine the degree k 2 of the fuzzy circle.

Optionally, the ranging points of the lidar evenly arranged around the monocular camera all fall on the diagonal of the image, and the ranging points are evenly distributed on the diagonal.

Optionally, the ranging points of the lidar evenly arranged around the monocular camera all fall on the center line of the image, and the ranging points are evenly distributed on the center line.

The present invention provides a system for correcting monocular ranging by using a single-line laser radar, including a ranging neural network model, a monocular camera and a plurality of laser radars; the optical axis of the monocular camera and the ray direction of each of the laser radars arranged in parallel; at least one of the lidars is arranged close to the monocular camera, and the rest of the lidars are evenly arranged around the monocular camera; the ranging neural network model is used to determine the inputted monocular camera to collect images The distance information of the target in the distance measurement neural network model is obtained by training the training data set and the global reference error matrix determined by the plurality of lidar ranging, and the training data set includes a plurality of images and each of the images. The actual distance in pixels.

Optionally, the ranging points of the lidar evenly arranged around the monocular camera all fall on the diagonal of the image, and the ranging points are evenly distributed on the diagonal; or, The ranging points of the lidar evenly arranged around the monocular camera all fall on the center line of the image, and the ranging points are evenly distributed on the center line.

The invention adopts the combination of laser radar and monocular camera, and uses multiple laser radars to provide multi-point accurate distance information to correct the global error of monocular ranging, which can simplify the algorithm difficulty of ranging, and at the same time improve the accuracy and reduce the cost. And the integration with various monocular ranging algorithms is good.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a schematic diagram of the principle of a defocus ranging algorithm provided by an embodiment of the present invention;

2 is a schematic flowchart of a method for correcting monocular ranging by using a single-line laser radar according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a training process of a monocular ranging correction neural network model provided by an embodiment of the present invention.

Detailed ways

In order to make the above objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

In the embodiment of the present invention, the distance information of the target is mainly obtained by a monocular camera. At present, the accuracy of the monocular ranging algorithm is generally not high. In the embodiment of the present invention, a single-line laser radar is used to correct the monocular camera to collect the depth value of a specific pixel point in a two-dimensional image, and then the correction information of a single pixel point is used to correct the entire image. Thereby, the accuracy of monocular ranging is improved.

It should be noted that the ranging error of each point in the monocular ranging algorithm is internally related. Taking the defocus ranging algorithm as an example: the defocus 3D measurement directly uses the relationship between the depth of the object, the parameters of the camera lens, and the blurriness of the image to measure the depth of the object.

FIG. 1 shows a schematic diagram of the principle of the defocus ranging algorithm. When the image point and the imaging plane c do not coincide, the object point a on the imaging plane c does not form a clear image point, but a blurred light spot with the same shape as the aperture diaphragm. For a general defocus ranging system, the aperture light The column has circular symmetry, so this blurred spot is a circular spot. As shown in Fig. 1, the radii R ₁ and R ₂ of the above-mentioned blurred light spot are related to the optical parameters of the optical system (such as the lens aperture D, the distances s ₁ and s ₂ from the imaging surface c to the lens b) and the object distance u of the object point a. There is the following relationship between:

(1)

(1)

(2)

(2)

(3)

(3)

When the optical parameters of the system are known, as long as the radius of the blurred light spot at a certain point on the object is measured, the object distance u of the point can be calculated. Therefore, it can be considered that the monocular ranging is a definite function at this time:

(4)

(4)

where L is an optical parameter.

But the optical parameters have some definite but complex effect on R ₁ , R ₂ :

(5)

(5)

τ is the algorithm for solving the blur radius based on the camera image:

(6)

(6)

That is, the algorithm will be affected by the actual object distance and some uncertain factors.

Therefore, each ranging error should have a definite relationship with the actual object distance, optical parameters, etc., but it is very complicated and it is difficult to obtain its analytical formula. It is precisely because the distances of each point are obtained from the same relationship, there must be an internal correlation between the errors of each point obtained in this way. If this correlation can be used, the error of one point can be reversed from the error of other points, thereby improving the accuracy of monocular ranging.

Neural networks happen to be able to establish correlations that are not easy to find. To illustrate how this neural network works, first describe the individual data and how they are acquired: a simple case is that the main optical axis of the monocular camera and the single-line lidar are on the same line, so that the distance measured by the lidar is always is the distance of the pixel from the center of the camera image. Using the defocus ranging algorithm, the predicted distance matrix S ₁ of the object in the picture can be obtained:

S ₁ =[ [s ₁₁ ,s ₁₂ ,s ₁₃ ,…]

[s ₂₁ ,s ₂₂ ,s ₂₃ ,…]

[…]

In order to get the label of the training set, it is also necessary to measure the real distance of the object in the image from the camera to form a label matrix Y, and then a neural network model can be established and trained. After the training is completed, the model can be used to perform target ranging on the images collected by the monocular camera.

Referring to the schematic flowchart of a method for correcting monocular ranging by using a single-line lidar shown in FIG. 2 , it is applied to a ranging system including a monocular camera and multiple lidars. The monocular camera and multiple lidars are preset Location arrangement, the method includes the following steps:

S202, an image collected by a monocular camera is acquired.

S204, input the image into a pre-trained ranging neural network model to obtain distance information of the target in the image.

The ranging neural network model is obtained by training a training data set, a global reference error matrix determined by multiple lidar ranging, and the training data set includes multiple images and the actual distances of pixels in each image.

The monocular camera and multiple lidars need to be arranged in preset positions, so that the ranging results of the monocular camera can be corrected by the ranging results of the lidars. In order to improve the correction effect and take into account the consideration of cost, this embodiment provides a specific arrangement as follows: the optical axis of the monocular camera is arranged in parallel with the ray direction of each lidar; at least one lidar is arranged close to the monocular camera, and the rest of the lidars are arranged around the monocular camera. The monocular cameras are evenly arranged.

Among them, the ranging point of the lidar arranged close to the monocular camera is as close as possible to the center point of the defocused image obtained by the monocular camera. Optionally, the ranging points of the lidar evenly arranged around the monocular camera all fall on the diagonal of the image, and the ranging points are evenly distributed on the diagonal; or, the lasers evenly arranged around the monocular camera The ranging points of the radar all fall on the center line of the image, and the ranging points are evenly distributed on the center line.

The global reference error matrix of the monocular camera ranging is determined through multiple lidar ranging, and the matrix is coupled with the pixel feature matrix of the image and then input to the distance neural network model for training, which can compare the actual object distance, optical parameters and other characteristic parameters with The relationship between the errors is regressed and fitted, and a distance neural network model capable of accurate ranging is obtained.

The method for correcting monocular ranging by using a single-line laser radar provided by the embodiment of the present invention adopts a combination of laser radar and a monocular camera, and uses multiple laser radars to provide accurate distance information of multiple points to correct the global error of monocular ranging , which can simplify the difficulty of the ranging algorithm, improve the accuracy, reduce the cost, and has good integration with various monocular ranging algorithms.

Optionally, the training process of the ranging neural network model is as follows:

First, the pixel features of each image in the training data set are extracted; then, the pixel feature correspondence matrix and the global reference error matrix of the image are coupled and input into the ranging neural network model for training until the loss function satisfies the preset termination condition.

The pixel features of the image include the position matrix of each pixel and the fuzzy circle size matrix. The position matrix represents the distance between each pixel and the center point. The fuzzy circle size matrix represents the fuzzy circle of each fuzzy circle relative to the center point or the fuzzy circle close to the center point. The ratio of the loss function is the difference between the model predicted distance and the actual distance.

The input of the ranging neural network model includes: the original image, the monocular camera distance prediction matrix A ₁ , and the original correction matrix B; the output is: the final distance prediction matrix A'.

Among them, the information provided by the original image includes the position of the pixel point, the size of the fuzzy circle, the overall target, etc. The relative position, the relative size of the fuzzy circle, etc. are used as the basis for correcting the correction matrix B, that is, the corrected correction matrix:

B'=f(position, fuzzy circle, B) (7)

The relationship between the above input and output is:

A'=A ₁ +B' (8)

If a real distance matrix A is obtained by actual measurement, then B'=AA ₁ is the desired optimized correction matrix. Therefore, B'-B can be used as the objective function.

Specifically, the coupling formula of the above-mentioned pixel feature corresponding matrix and the global reference error matrix of the image is as follows:

(9)

(9)

Among them, L _n is the distance error matrix after coupling, γ is the correction coefficient, L ₀ is the global reference error matrix determined by multiple lidar ranging, S _n is the matrix composed of the distance between all pixels in the image and the center point of the image, On is a matrix composed of the ratio of any fuzzy circle in the image to the fuzzy circle at the center of the image or the fuzzy circle close to the center of the image, k 1 and k 2 are the times of pre-calibration, and _⨀ represents the element product operation. In the coupling formula, the degree k 2 of the fuzzy circle size matrix and the degree k 1 of the relative position matrix are obtained by calibration.

If only the relationship between the relative position and the ranging error is considered, let L ₀ be the monocular ranging error measured by the laser ranging point (for the sake of simplicity, only the case where the laser ranging point is the center of the camera pixel is considered) , from the prior knowledge, it can be known that any pixel _point has the following relationship with the distance Sn from the center point:

(10)

(10)

Among them, L _n is the pixel point corresponding to _Sn ; α is the correction coefficient; because it is considered that the error of the center point is 0 after subtracting the error of the lidar ranging, so there is no offset, and ⨀ represents the element product operation.

In the same way, let the size of any fuzzy circle relative to the center or the fuzzy circle close to the center be O _n = fuzzy circle at this point/fuzzy circle near the center point, then if only the size of the fuzzy circle is considered, there are:

(11)

(11)

Among them, L _n is the pixel point corresponding to _Sn ; β is the correction coefficient.

Assuming that the fuzzy circle and the pixel position are independent of each other, there are:

Multiply these three matrices according to the above formula, put them into the fully connected neural network for training, and optimize the actual error measured by the experiment, so that the final result converges to the actual distance matrix, and the trained monocular ranging correction neural network is obtained. network model.

，耦合后的原始校正矩阵与位置矩阵

，一起作为全连接神经网络的输入，神经网络的输出为校正后的深度图，将校正后的深度图与实际测得的深度图进行比较，作为损失函数通过梯度下降算法改变神经网络的权重，直至损失函数结果可接受。Referring to the schematic diagram of the training process of the monocular ranging correction neural network model shown in Figure 3, the fuzzy circle size matrix is obtained by applying the image captured by the monocular camera to the relevant optical formula

, the coupled original correction matrix and position matrix

, together as the input of the fully connected neural network, the output of the neural network is the corrected depth map, the corrected depth map is compared with the actual measured depth map, and the weight of the neural network is changed as a loss function through the gradient descent algorithm, until the loss function result is acceptable.

In practical applications, in order to further improve the correction effect and consider factors such as cost, this embodiment exemplarily proposes an improved method using an arrangement of one camera and five lidars and a coupling formula.

The main optical axis of the monocular camera is arranged in parallel with the ray directions of the five single-line lidars, ensuring that the pixel positions of the lidar ranging points in the images obtained by the monocular camera are basically fixed. One of the lidars is fixed close to the monocular camera, so that the laser ranging point is as close as possible to the center point of the defocused image obtained by the camera, and the other four lidars are slightly away from the monocular camera and evenly arranged around the camera, for example, four laser ranging Points 3 equally divide the 2 diagonals of the image. The position of the pixel point of the camera image corresponding to the laser ranging point is basically located at the center of the entire image and at the third point of the two diagonal lines.

For the improvement of the coupling formula, the error calculated by the distance measurement of the lidar close to the center point and the monocular distance measurement is used as the global reference error L ₀ ', and the image is divided into upper left, upper right, lower left and lower right according to the pixel position. The errors of the monocular ranging calculated by the Lidar ranging from the four distances far from the camera are used as the local reference errors L ₁₁ , L ₁₂ , L ₂₁ , and L ₂₂ for the above four parts. Fill L ₁₁ , L ₁₂ , L ₂₁ , and L ₂₂ to the corresponding positions of the error matrix according to the number of pixels to obtain the local error matrix L ₁ , namely:

L ₁ =[[ L ₁₁ L ₁₂ ]

[ L ₂₁ L ₂₂ ]]

Specifically, the error between the ranging result of the lidar close to the monocular camera and the ranging result of the monocular camera is taken as the global reference error L ₀ ′ , and the ranging result of the lidar evenly arranged around the monocular camera is compared with the monocular camera. The error of the ranging result is taken as the local reference error L ₁ , L ₀ is as follows:

L ₀ = L ₀ '+ λ∙L ₁

Among them, λ is the influence coefficient of global reference error and local reference error.

Optionally, the above method also includes a calibration step of the number k 2 of the size matrix of the fuzzy circle and the number of k 1 of the relative position matrix, because the cameras are different, the number of times of the fuzzy circle, position, etc. should also be different, and can be calibrated by the following method: The times:

Perform polynomial fitting on the errors at different positions measured by a straight line along the radial direction to determine the relative position order k 1; perform multi-form fitting on the errors at different positions measured on a straight line along the axial direction to determine the fuzzy circle The number of times k 2.

Place the test object at different positions in the axial and radial directions of the camera to obtain the relationship data between the error and the fuzzy circle and the pixel position, fit the relationship between the size, position and error of the fuzzy circle by a polynomial, and separately calibrate the size of the fuzzy circle O _n and the pixel _Number of times for position Sn .

The method of checking errors in this embodiment is also applicable to other monocular ranging algorithms. If the laser radar and the camera are arranged at a certain angle, the matching of the laser ranging points and the camera pixels should be performed.

The embodiment of the present invention also provides a system for correcting monocular ranging by using a single-line laser radar, which can implement the method for correcting monocular ranging by using a single-line laser radar provided in the above-mentioned embodiments, which may specifically include a ranging neural network model, a single-lens camera and multiple lidars.

Among them, the optical axis of the monocular camera is arranged parallel to the ray direction of each lidar; at least one lidar is arranged close to the monocular camera, and the rest of the lidars are evenly arranged around the monocular camera; the ranging neural network model is used to determine the input monocular The camera collects the distance information of the target in the image, and the ranging neural network model is trained from the training data set and the global reference error matrix determined by multiple lidar ranging. The training data set includes multiple images and the actual distance of the pixels in each image. .

The system for correcting monocular ranging by using a single-line laser radar provided by the embodiment of the present invention adopts a combination of laser radar and a monocular camera, and uses multiple laser radars to provide accurate distance information of multiple points to correct the global error of monocular ranging , which can simplify the algorithm difficulty of ranging, while improving the accuracy and reducing the cost.

Optionally, the ranging points of the lidar evenly arranged around the monocular camera all fall on the diagonal of the image, and the ranging points are evenly distributed on the diagonal; or, the lasers evenly arranged around the monocular camera The ranging points of the radar all fall on the center line of the image, and the ranging points are evenly distributed on the center line.

Those skilled in the art can understand that the realization of all or part of the processes in the methods of the above embodiments can be accomplished by instructing the control device through a computer level, and the program can be stored in a computer-readable storage medium. During execution, the processes of the above-mentioned method embodiments may be included, wherein the storage medium may be a memory, a magnetic disk, an optical disk, or the like.

In this document, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such existence between these entities or operations. The actual relationship or sequence. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. a method that utilizes single-line laser radar to correct monocular ranging, is characterized in that, is applied to the ranging system comprising monocular camera and multiple laser radars, and described monocular camera and described multiple laser radars are pre- Setting the location arrangement, the method includes: acquiring an image captured by the monocular camera; Input the image into a pre-trained ranging neural network model to obtain the distance information of the target in the image; the ranging neural network model is determined by the training data set and the global reference error matrix of the multiple lidar ranging It is obtained through training that the training data set includes a plurality of images and the actual distances of pixels in each of the images. 2. method according to claim 1, is characterized in that, the training process of described ranging neural network model is as follows: extracting pixel features of each image in the training data set; The pixel feature corresponding matrix and the global reference error matrix of the image are coupled and then input into a ranging neural network model for training until the loss function satisfies a preset termination condition. 3. The method according to claim 1, wherein the optical axis of the monocular camera is arranged in parallel with the ray direction of each of the lidars; At least one of the lidars is arranged close to the monocular camera, and the rest of the lidars are evenly arranged around the monocular camera. 4. method according to claim 1, is characterized in that, described pixel characteristic correspondence matrix, the global reference error matrix coupling formula of described image is as follows:

Wherein, L _n is the distance error matrix after coupling, γ is the correction coefficient, L ₀ is the global reference error matrix determined by the multiple lidar ranging, and S _n is the distance between all pixels in the image and the center point of the image. matrix, O _n is the matrix composed of the ratio of any fuzzy circle in the image to the fuzzy circle at the center of the image or the fuzzy circle close to the center of the image, k 1 and k 2 are the pre-calibrated times, and ⨀ represents the element-wise product operation. 5 . The method according to claim 4 , wherein the error between the ranging result of the lidar close to the monocular camera and the ranging result of the monocular camera is used as the global reference error L ₀ ′ , and the surrounding The error between the ranging result of the lidar with the monocular cameras evenly arranged and the ranging result of the monocular camera is taken as the local reference error L ₁ , and L ₀ is as follows:

Among them, λ is the influence coefficient of global reference error and local reference error. 6. The method according to claim 4, wherein the method further comprises: Perform polynomial fitting on the errors of different positions measured along a straight line in the radial direction to determine the relative position order k 1; Perform multi-form fitting on the errors at different positions measured by a straight line along the axial direction to determine the number of fuzzy circles k 2 . 7 . The method according to claim 3 , wherein the ranging points of the lidar evenly arranged around the monocular camera all fall on the diagonal line of the image, and each ranging point is on the diagonal line of the image. 8 . The diagonals are evenly distributed. 8 . The method according to claim 3 , wherein the ranging points of the lidar evenly arranged around the monocular camera all fall on the center line of the image, and each ranging point is on the center line of the image. 9 . Evenly distributed on the midline. 9. A system for correcting monocular ranging by single-line lidar, characterized in that it comprises a ranging neural network model, a monocular camera and a plurality of lidars; The optical axis of the monocular camera is arranged in parallel with the ray direction of each of the lidars; At least one of the lidars is arranged close to the monocular camera, and the rest of the lidars are evenly arranged around the monocular camera; The ranging neural network model is used to determine the distance information of the target in the input image collected by the monocular camera, and the ranging neural network model is based on the training data set and the global reference error determined by the plurality of lidar ranging. Matrix training is obtained, and the training data set includes a plurality of images and the actual distances of pixels in each of the images. 10 . The system according to claim 9 , wherein the ranging points of the lidar evenly arranged around the monocular camera all fall on the diagonal line of the image, and each ranging point is on the diagonal line of the image. 10 . evenly distributed on the diagonal; or, The ranging points of the lidar evenly arranged around the monocular camera all fall on the center line of the image, and the ranging points are evenly distributed on the center line.

Images