CN109035329A - Camera Attitude estimation optimization method based on depth characteristic - Google Patents

Abstract

The camera pose estimation optimization method based on depth features relates to the optimization method based on supervised learning SLAM system. Use the random forest-based matching algorithm to quickly calculate the similarity of 2D-3D points to map 2D-3D point information; use the method of constraint function and multi-feature fusion to evaluate the camera pose; aim at the instability existing in the SLAM algorithm based on deep learning A multi-feature bundle optimization algorithm is proposed. Using the 3D reconstruction data as a reference, we then use the visible 3D points and associated keypoint maps from the random forest offline dataset, and use multi-feature fusion and constraint functions to measure the pose evaluation score. The above methods are used to optimize the performance of deep learning-based SLAM. Experimental results prove that the algorithm is robust.

Description

Optimal Method for Camera Pose Estimation Based on Depth Features

technical field

The present invention relates to an optimization method based on a supervised learning SLAM system, in particular to a depth feature-based camera pose estimation optimization method for a supervised learning SLAM algorithm.

Background technique

SLAM technology has good application prospects in the fields of robotics, autonomous driving, virtual and augmented reality. Among many computer vision and artificial intelligence technologies, SLAM research continues to be hot. In recent years, more and more robots have appeared in people's field of vision, bringing a lot of convenience to people's lives. Through their own cameras, gyroscopes, laser sensors, etc., they can obtain the environment of specific scenes and position themselves. Complete specific tasks under real-time conditions. In recent years, many companies at home and abroad have invested a lot of manpower and material resources in the research and development of unmanned vehicles. The core technology of unmanned driving is also SLAM technology. Robust and fast environment recognition and semantic segmentation are the key to this unmanned driving. In the field of augmented reality, most of the AR applications currently on the market in commercial scenarios are based on specific templates, from template recognition to template tracking and matching, combined with 3D registration and model rendering for virtual-real interaction. The real augmented reality requires recognition and semantic understanding of the environment where the application scene is located. At this time, SLAM technology is still needed as the core technology.

Camera relocalization has always been an important task in SLAM. Image-based relocalization methods are a powerful and effective thread in SLAM. Common techniques for imaging-based camera pose estimation are image retrieval and methods based on 3D scene reconstruction. However, image retrieval errors are larger than GPS position sensors. In addition, the estimation accuracy also depends on the data set, ARZamir et al. 2014)) proposed a multiple nearest neighbor geolocation feature matching framework, but this method is severely limited if the query image (Query) and the dataset (Database) do not match. This approach requires 3D prior information to provide valuable spatial relationships in order to achieve better localization results. However, existing image localization methods based on 3D priors only focus on local localization accuracy and ignore subsequent optimization. For local optimization, such as FastSLAM ([2] Parsons, S.: Probabilistic robotics by Sebastian Thrun, Wolfram Burgard and Dieter Fox. Knowledge Eng. Review (2006). https://doi.org/10.1017/S0269888906210993), by sensor estimation noise to measure the estimated pose error and optimize it. ORB-SLAM ([3] Mur-Artal, R., Montiel, J.M.M., Tard′os, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM System.IEEE Trans.Robot.(2015)) is an excellent visual SLAM system, using orb features and local and global optimization, but it does not use deep features and prior knowledge. ([4]Kendall, A., Cipolla, R.: Modeling uncertainty in deep learning for camera localization. In: IEEE International Conference on Robotics and Automation, ICRA 2016, Stockholm, Sweden, 16–21 May 2016. https://doi. org/10.1109/ICRA.2016.7487679) proposed a visual relocalization system using Bayesian and convolutional neural networks to regress camera poses, reusing 3D information and approaching real-time performance. Our proposed method does not utilize any sensors, but is based on a priori 3D point similarity localization model, however, none of the current methods based on supervised learning have quite applicable optimization methods. Although some visual localization algorithms based on deep convolutional neural networks are considered as end-to-end localization methods that tolerate large baselines, the average error often occurs when estimating poses. It all boils down to no back-end optimization (e.g. local bundle optimization), based on the above problem, shows how to use random forests (Cutler,A.,Cutler,D.R.,Stevens,J.R.:Random forests.In:Machine Learning(2012)) Do 2D-3D point matching, and how to improve and use constraint functions to optimize SLAM systems based on deep learning.

Contents of the invention

The object of the present invention is to provide a camera pose estimation optimization method based on depth features.

The present invention comprises the following steps:

1) Use the random forest-based matching algorithm to quickly calculate the similarity of 2D-3D points to map 2D-3D point information;

2) Evaluate the camera pose using constraint functions and multi-feature fusion;

3) Aiming at the instability problem existing in the SLAM algorithm based on deep learning, a multi-feature bundle optimization algorithm is proposed.

In step 1), the specific method for quickly calculating the similarity of 2D-3D points to map 2D-3D point information using a random forest-based matching algorithm can be:

Each decision tree is composed of internal nodes and leaf nodes. The prediction of the decision tree can calculate the similarity between 2D pixel points, and then calculate the similarity between 2D pixel points to the similarity in three-dimensional space, starting from the root node Up to the leaves, the training converges by repeatedly modifying the decision function. The decision function is expressed as follows:

split(p; δ _n ) = [f _n (p) > δ _n ]

Among them, n represents the index of the node in the decision tree, p is the non-leaf node representing the 2D key point, [.] is the index function from 0 to 1, δ is the decision threshold, and f() is the decision function:

f(p)＝a ₁ D _shape (p ₁ ,p ₂ )+a ₂ D _texture (p ₁ ,p ₂ )+a ₃ D _color (p ₁ ,p ₂ )

Define a and D() in feature fusion. If split(p; δn) evaluates to 0, it is the training path branching to the left subtree, otherwise branching to the right, p1 and p2 are the pixel pairs around the key point , in the process of training, the 3D reconstruction data contains the mapping of the corresponding 3D points and 2D points; the prior is divided into training data and verification data; the training framework is shown in Algorithm 1; the objective function Q is used to make the training data and The verification data has the same learning tendency, where Θ is the number of verification data entering different paths from the related training data, P _verification is the verification data set, and λ is a parameter that compromises the similarity of the same branch and the diversity of different branches; the objective function is based on the proportion of validation data and training data that fall into the same branch to measure the similarity between points:

Algorithm 1 The training process of 2D-3D point mapping based on random forest is shown in Table 1:

Table 1

In step 2), the specific method for evaluating the camera pose using the constraint function and multi-feature fusion method can be:

Suppose the pose estimation result is the camera extrinsic parameters It is converted from x, y, z (position information) and w, p, q, r (four elements), assuming that the internal reference matrix K is obtained from the EXIF tag (assuming no radial deformation); then, combined with the internal reference:

The transformation matrix can be obtained by image coordinates and world coordinates:

2D points and 3D point clouds are associated through this transformation matrix; for example, extracting the FALoG feature points of an evaluation image that has already evaluated the camera matrix as a query data set. For the query feature point m(x, y), the most approximate feature of the query feature can be found through the above mapping algorithm, and the nearest established data set feature can be searched. The random forest tests the query feature in each decision node, and finally the query reaches Leaf node; the final mapping feature m' of the random forest is the corresponding point with the highest probability of the node at point m, and the corresponding point m' has its associated 3D point. For each 3D point, the database has at least one corresponding image, each point can be projected to the relevant image, combined with the pixel difference as the error function, and using the error function to iteratively optimize the pose matrix, it is considered that the error of different mapping points can be obtained by Color-based, shape-based and texture-based pixel-level error representations, for each feature using a score that evaluates the pose with the highest deviation rate based on the above features;

The relevant characteristics are as follows:

(1) Color-based pixel features.

The color feature can represent the surface properties of the object in the image, and the fusion pixel feature method of fusing the single pixel and the image area can express the color feature of a single pixel to the greatest extent; use the color distance function to distinguish two single pixel color changes:

Among them, p(x, y) and p'(x, y) are two target points, R(), G() and B() are the respective corresponding R, G, and B values; for the regional color feature, Suppose the target point contained in an area, assuming there are 5×5 pixels, for each RGB channel, the center of the area is the target point; extract the gradient value G( x,y):

where, d _h is the average value of the difference target point between the two pixels on the left and the average value of the two pixels on the right; the center value 48 and the other values in the horizontal and vertical areas are d _v , calculated in the same way by d _h owned. For the remaining four 2×2 pixel blocks, average downsampling, finally, the color-based pixel disparity is:

Among them, δ∈{0,1}, if the difference between the downsampled values of two blocks is lower than d _point (p,p'), then δ is set to 0, otherwise it is 1; d _region (p,p') adopts The same calculation method as d _point (p,p'), d _G (p,p') is the difference of the gradient value, and η is the regular factor;

(2) Shape-based and texture-based pixel features

As the center of the 32×32 block, the shape context features are extracted, and the shape context of 25 sampling points is calculated, and the spatial logarithmic coordinates of the distribution block are divided into 12×3=36 parts, and the difference between the shape features is expressed using binarization. point shape diversity, Among them, n=36, item ∈ {0,1}, item=1 means that the column value of block i is greater than the average value, for texture, the center point of a circle with radius R is divided into eight equal angle regions; calculate The average image intensity of each area, if the average value is greater than the central pixel value, the value of the area is set to 1, otherwise, it is set to 0; eight binary sequences are converted to decimal numbers to represent texture features; finally, D _texture is represented by The Hamming distance is determined;

Finally, the evaluation function E is a combination of the above features:

E(p _i ,p _j )＝a ₁ D _shape (p _i ,p _j )+a ₂ D _texture (p _i ,p _j )+a ₃ ||D _color (p _i ,p _j )|| ₂

Among them, a _i ∈ (0,1) represents the weight of different items, D _color is regarded as a regular item in this fused diversity function, D _shape and D _texture are connected, and training through k iterations makes this function The parameters of can be evaluated best, and the evaluation function describes that the pose evaluation error can be measured by the reprojection point error of the query 2D feature and the prior dataset feature on the feature point.

In step 3), the specific method of proposing a multi-feature cluster optimization algorithm for the instability problem existing in the SLAM algorithm based on deep learning is as follows:

A deep learning SLAM system based on prior knowledge stores 2D-3D maps and keyframes by opening a new thread, and selects those that satisfy The key frame of the key frame is used as the key frame set of bundle adjustment (BA for short); the local BA optimizes all points that can be seen by the key frame; the observation point helps to optimize the constraint function and the final pose, global BA optimization and local BA Optimization is similar, except that the system will perform local BA between 30 frames and global BA between 300 frames; the comparison between the SLAM system using the optimized algorithm and other advanced SLAM systems is shown in Table 2.

Table 2

The present invention combines the SLAM technology based on deep learning and the mapping between two-dimensional image points and three-dimensional point clouds based on random forests, and designs a SLAM system optimization algorithm based on deep learning, which well solves the gap in the deep learning SLAM system optimization algorithm. The system integrates low-calculation SLAM environment construction, image 2D point and 3D point cloud matching, and can reconstruct and optimize the scene in real time on both the PC and mobile terminals, maintaining a relatively high reconstruction accuracy. Human driving, augmented reality and other fields have important practical value and significance.

Most existing localization methods only approximate pose confidence based on reference point distance. Different from other methods, the present invention uses 3D reconstruction data as reference, then uses visible 3D points and related keypoint mapping from random forest offline dataset, and uses multi-feature fusion and constraint function to measure pose evaluation score. The above methods are used to optimize the performance of deep learning-based SLAM. Experimental results prove that the algorithm of the present invention is robust.

Description of drawings

Figure 1 is a system overview of 2D-3D point matching.

Figure 2 is an illustration of a 5x5 pixel region based on color features.

Figure 3 is an illustration of shape-based features.

Detailed ways

The following embodiments will further illustrate the present invention in conjunction with the accompanying drawings.

1. Basic concepts

1) 2D-3D point mapping based on random forest

The purpose of constructing a random forest is to find the relationship between the query image and the 3D reconstruction prior data. The pose of different localization algorithms can be evaluated through this mapping architecture. Because the random forest algorithm has the advantages of lower complexity and higher robustness, the mapping algorithm meets real-time and effective evaluation. In essence, a decision tree is just a mapping tool for retrieving triangular geometric relations from 2D to 3D, which can be replaced by other methods.

The training of the decision tree is the key to the mapping algorithm. The performance of random forest is determined by the integration performance of various decision trees. Each decision tree consists of internal nodes and leaf nodes. The prediction of the decision tree can calculate the similarity between 2D pixel points, and then calculate the similarity between 2D pixel points to the similarity in three-dimensional space. From the root node to the leaves, the training converges by repeatedly modifying the decision function. The decision function is expressed as follows:

split(p; δ _n ) = [f _n (p) > δ _n ]

Among them, n represents the index of the node in the decision tree, p is the non-leaf node representing the 2D key point, [.] is the 0-1 index function, δ is the decision threshold, and f() is the decision function:

f(p)＝a ₁ D _shape (p ₁ ,p ₂ )+a ₂ D _texture (p ₁ ,p ₂ )+a ₃ D _color (p ₁ ,p ₂ )

Define a and D() in feature fusion. If split(p; δn) evaluates to 0, the training path branches to the left subtree, otherwise it branches to the right. p1 and p2 are pixel pairs around the keypoint. During training, the 3D reconstruction data (prior data) contains the corresponding 3D points and the mapping of 2D points. Divide the prior into training and validation data. The training framework is shown in Algorithm 1. It should be noted that the fast FALoG feature points are used (Wang, Z., Fan, B., Wu, F.: FRIF: fast robust invariant feature. In: British MachineVision Conference, BMVC2013, Bristol, UK, 9–13 September 2013 .https://doi.org/10.5244/C.27.16) can quickly detect the corresponding binary feature points. The objective function Q is used to make the training data and validation data have the same learning tendency as much as possible. where Θ is the number of validation data that enters a different path from the associated training data, and P _verification is the validation dataset. λ is a parameter that trades off the similarity of the same branch and the diversity of different branches. The objective function is to measure the similarity between points based on the ratio of validation data and training data falling into the same branch.

Algorithm 1 The training process of 2D-3D point mapping based on random forest is shown in Table 1.

2) Feature Fusion Algorithm

Although the relationship of 2D and 3D is easily obtained by the above mapping. Since there is no image pose label, it is a difficult metric to judge whether the camera pose is good or bad. Design a set of camera pose evaluation algorithm (evaluate the score of camera pose), this algorithm does not require manual labeling, but only needs 3D reconstruction data obtained through transfer learning. The camera pose to be predicted can be computed by any camera pose prediction algorithm. Suppose the pose estimation result is the camera extrinsic parameters It is converted from x, y, z (position information) and w, p, q, r (four elements). Assume that the internal reference matrix K is obtained from EXIF tags (assuming no radial deformation). Then, combined with internal parameters:

The transformation matrix can be obtained by image coordinates and world coordinates:

2D points and 3D point clouds are related through this transformation matrix. For example, extracting the FALoG feature points of an evaluation image that has evaluated the camera matrix as a query data set. For the query feature point m(x, y), the most approximate feature of the query feature can be found through the above mapping algorithm. As shown in Figure 1, searching for the nearest dataset features that have been established, Random Forest passes through testing the query features in each decision node, and finally the query reaches the leaf nodes. The final mapping feature m' of the random forest is the corresponding point with the highest node probability at point m. The corresponding point m' has its associated 3D point. For each 3D point, the database has at least one corresponding image, and each point can be projected onto the associated image. Combine the pixel difference as an error function, and use this error function to iteratively optimize the pose matrix. It is considered that the errors of different mapped points can be represented by color-based, shape-based and texture-based pixel-level errors. For each feature, we describe it below. Finally, a score is used to evaluate the pose with the highest deviation rate based on the above features.

1. Color-based pixel features.

Color features can represent the surface properties of objects in an image. In the present invention, the color feature of a single pixel can be expressed to the greatest extent by adopting a fusion pixel feature method that fuses a single pixel and an image region. Use a color distance function to distinguish between two single pixel color changes:

where p(x,y) and p'(x,y) are two target points, and R(), G(), and B() are the respective corresponding R, G, and B values. For the region color feature, it is assumed that the target point is contained in a region, assuming that there are 5×5 pixels, as shown in Figure 2, for each RGB channel, the center of the region is the target point. Extract the gradient value G(x, y) in the horizontal and vertical directions of the target point (x, y)

where d _h is the mean of the difference target point between the two pixels on the left and the mean of the two pixels on the right. The other values in the horizontal and vertical regions of the central value 48 in FIG. 2 are d _v , calculated in the same way as d _h in FIG. 2 . For the remaining four 2×2 pixel blocks (points in Figure 2 except the center point vertical and horizontal regions), average their downsampling. Finally, the pixel disparity based on color is:

Among them, δ ∈ {0,1}, if the difference of the downsampled values of two blocks is lower than d _point (p,p'), then δ is set to 0, otherwise it is 1. d _region (p,p') uses the same calculation method as d _point (p,p'). d _G (p,p') is the difference of the gradient value, η is the regular

factor.

2. Shape-based and texture-based pixel features.

In order to extract the shape feature point of the pixel faster and more efficiently, as the center of the 32×32 block, the shape context feature is extracted, and the shape context of 25 sampling points is calculated. As shown in FIG. 3, the spatial logarithmic coordinates of the allocation block are divided into 12×3=36 parts. Histograms are used to represent shape feature vectors. In order to express the difference in shape features, binarization is used to represent the diversity of the transformed two-point shape histogram, Where n = 36, item ∈ {0,1}, item = 1 means that the column value of block i is greater than the average value of the histogram. For textures, the center point of a circle of radius R is divided into eight equally angular regions. Calculate the average image intensity of each region, if the average value is greater than the center pixel value, the value of this region is set to 1, otherwise, it is set to 0. Eight binary sequences are converted to represent texture features as decimal numbers. Finally, the D _texture is determined by the Hamming distance.

Finally, the evaluation function E is a combination of the above features:

E(p _i ,p _j )＝a ₁ D _shape (p _i ,p _j )+a ₂ D _texture (p _i ,p _j )+a ₃ ||D _color (p _i ,p _j )|| ₂

Among them, a _i ∈ (0,1) represents the weight of different items. D _color is regarded as a regularization term in this fused diversity function. D _shape and D _texture are connected. The parameters in this function can be best evaluated by training for k iterations. The evaluation function describes that the pose estimation error can be measured by the reprojection point error of the query 2D features and the prior dataset features on the feature points.

3) Backend optimization based on deep SLAM

Local localization errors will be reflected in pose prediction and 3D reconstruction between different keyframes. Accumulation of keyframe pose estimation errors leads to global error growth, such that the accuracy of the entire system is limited. Deep learning SLAM systems based on prior knowledge store 2D-3D maps and keyframes by opening a new thread. choose those that meet The key frames of are used as the key frame set of bundle adjustment (BA for short). Local BA optimizes all points that can be seen by keyframes. The observation points help in the optimization of the constraint function and the final pose. Global BA optimization is similar to local BA optimization, except that the system will perform local BA between 30 frames and global BA between 300 frames.

The comparison between the SLAM system using the optimization algorithm of the present invention and other advanced SLAM systems is shown in Table 2.

Claims (4)

1. The camera pose estimation optimization method based on depth feature is characterized in that comprising the following steps: 1) Use the random forest-based matching algorithm to quickly calculate the similarity of 2D-3D points to map 2D-3D point information; 2) Evaluate the camera pose using constraint functions and multi-feature fusion; 3) Aiming at the instability problem existing in the SLAM algorithm based on deep learning, a multi-feature bundle optimization algorithm is proposed. 2. the camera pose estimation optimization method based on depth feature as claimed in claim 1, is characterized in that in step 1), described use is based on the matching algorithm of random forest, the similarity of fast calculation 2D-3D point is to map 2D- The specific method of 3D point information is: Each decision tree is composed of internal nodes and leaf nodes. The prediction of the decision tree can calculate the similarity between 2D pixel points, and then calculate the similarity between 2D pixel points to the similarity in three-dimensional space, starting from the root node Up to the leaves, the training converges by repeatedly modifying the decision function. The decision function is expressed as follows: split(p; δ _n ) = [f _n (p) > δ _n ] Among them, n represents the index of the node in the decision tree, p is the non-leaf node representing the 2D key point, [.] is the index function from 0 to 1, δ is the decision threshold, and f() is the decision function: f(p)＝a ₁ D _shape (p ₁ ,p ₂ )+a ₂ D _texture (p ₁ ,p ₂ )+a ₃ D _color (p ₁ ,p ₂ ) Define a and D() in feature fusion. If split(p; δn) evaluates to 0, it is the training path branching to the left subtree, otherwise branching to the right, p1 and p2 are the pixel pairs around the key point , in the process of training, the 3D reconstruction data contains the mapping of the corresponding 3D points and 2D points; the prior is divided into training data and verification data; the training framework is shown in Algorithm 1; the objective function Q is used to make the training data and The verification data has the same learning tendency, where Θ is the number of verification data entering different paths from the related training data, P _verification is the verification data set, and λ is a parameter that compromises the similarity of the same branch and the diversity of different branches; the objective function is based on the proportion of validation data and training data that fall into the same branch to measure the similarity between points: Algorithm 1 The training process of 2D-3D point mapping based on random forest is shown in Table 1: Table 1 3. the camera pose estimation optimization method based on depth feature as claimed in claim 1, it is characterized in that in step 2) in, the concrete method of the method evaluation camera pose described using constraint function and multi-feature fusion is: Suppose the pose estimation result is the camera extrinsic parameters It is obtained by converting x, y, z and w, p, q, r, assuming that the internal reference matrix K is obtained from the EXIF tag; then, combined with the internal reference: Get the transformation matrix by image coordinates and world coordinates: 2D points and 3D point clouds are associated through this transformation matrix; extract a FALoG feature point of an evaluation image that has already evaluated the camera matrix as the query data set; for the query feature point m(x, y), find the query feature through the above mapping algorithm The most approximate feature, search for the most recent data set features that have been established, the random forest tests the query feature in each decision node, and finally the query reaches the leaf node; the final mapping feature m' of the random forest is the node with the highest probability at m points Corresponding points, the corresponding point m' has its associated 3D point; for each 3D point, the database has at least one corresponding image, each point is projected to the associated image, combine the pixel difference as an error function, and use the error function Iteratively optimize the pose matrix, considering that the errors of different mapping points are represented by color-based, shape-based and texture-based pixel-level errors, and for each feature, use the score based on the attitude with the highest deviation rate based on the above features; The relevant characteristics are as follows: (1) Color-based pixel features The color feature can represent the surface properties of the object in the image, and the fusion pixel feature method of fusing the single pixel and the image area is used to express the color feature of a single pixel to the greatest extent; the color distance function is used to distinguish two single pixel color changes: Among them, p(x, y) and p'(x, y) are two target points, R(), G() and B() are the respective corresponding R, G, and B values; for the regional color feature, Suppose the target point contained in an area, assuming there are 5×5 pixels, for each RGB channel, the center of the area is the target point; extract the gradient value G( x,y): where, d _h is the average value of the difference target point between the two pixels on the left and the average value of the two pixels on the right; the center value 48 and the other values in the horizontal and vertical areas are d _v , calculated in the same way by d _h Obtained; for the remaining four 2×2 pixel blocks, average downsampling, and finally, the color-based pixel dissimilarity is: Among them, δ∈{0,1}, if the difference between the downsampled values of two blocks is lower than d _point (p,p'), then δ is set to 0, otherwise it is 1; d _region (p,p') adopts The same calculation method as d _point (p,p'), d _G (p,p') is the difference of the gradient value, and η is the regular factor; (2) Shape-based and texture-based pixel features As the center of the 32×32 block, the shape context features are extracted, and the shape context of 25 sampling points is calculated, and the spatial logarithmic coordinates of the distribution block are divided into 12×3=36 parts, and the difference between the shape features is expressed using binarization. point shape diversity, Among them, n=36, item ∈ {0,1}, item=1 means that the column value of block i is greater than the average value, for texture, the center point of a circle with radius R is divided into eight equal angle regions; calculate The average image intensity of each area, if the average value is greater than the central pixel value, the value of the area is set to 1, otherwise, it is set to 0; eight binary sequences are converted to decimal numbers to represent texture features; finally, D _texture is represented by The Hamming distance is determined; Finally, the evaluation function E is a combination of the above features: E(p _i ,p _j )＝a ₁ D _shape (p _i ,p _j )+a ₂ D _texture (p _i ,p _j )+a ₃ ||D _color (p _i ,p _j )|| ₂ Among them, a _i ∈ (0,1) represents the weight of different items, D _color is regarded as a regular item in this fused diversity function, D _shape and D _texture are connected, and training through k iterations makes this function The parameters of can be evaluated best, and the evaluation function describes the pose evaluation error as measured by the reprojection point error of the query 2D feature and the prior dataset feature on the feature point. 4. as claimed in claim 1 based on the camera pose estimation optimization method of depth feature, it is characterized in that in step 3) in, the described instability problem existing in the SLAM algorithm based on deep learning, propose a kind of multi-feature bundle The specific method of the optimization algorithm is as follows: A deep learning SLAM system based on prior knowledge stores 2D-3D maps and keyframes by opening a new thread, and selects those that satisfy The key frame of the key frame is used as the key frame set of bundle adjustment (BA for short); the local BA optimizes all points seen by the key frame; the observation point contributes to the optimization of the constraint function and the final pose, global BA optimization and local BA optimization Similar, except that the system will perform local BA between 30 frames and global BA between 300 frames; Table 2 The comparison between the SLAM system using the optimized algorithm and other advanced SLAM systems is shown in Table 2.