WO2022205750A1 - Point cloud data generation method and apparatus, electronic device, and storage medium - Google Patents

Abstract

The present disclosure provides a point cloud data generation method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a scene image corresponding to a current scene collected by a target device and positioning pose information of the target device; extracting at least one two-dimensional feature points from the scene image by using a feature point extraction means corresponding to a simultaneous localization and mapping (SLAM) system; according to the positioning pose information and position information of the two-dimensional feature points in the scene image, determining, in a pre-constructed three-dimensional scene map, three-dimensional feature points which match the two-dimensional feature points and three-dimensional position information of the three-dimensional feature points; generating point cloud data corresponding to the current scene on the basis of the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points.

Description

Point cloud data generation method, device, electronic device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure is based on the Chinese patent application with the application number of 202110348215.2, the application date of March 31, 2021, and the application name of "point cloud data generation method, device, electronic device and storage medium", and requires the priority of the Chinese patent application The entire content of this Chinese patent application is incorporated herein by reference.

technical field

The embodiments of the present disclosure relate to the field of positioning technologies, and to a method, an apparatus, an electronic device, and a storage medium for generating point cloud data.

Background technique

Simultaneous localization and mapping (SLAM) refers to a mobile device moving from an unknown location in an unknown environment, positioning itself according to the location estimation and map during the movement process, and building on the basis of its own positioning. Incremental maps, enabling autonomous positioning and navigation of mobile devices. Generally, a SLAM system running on a mobile device will generate a large error accumulation in long-distance tracking, which reduces the accuracy and stability of the SLAM system.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present disclosure provide at least a method, apparatus, electronic device, and storage medium for generating point cloud data.

In one aspect, an embodiment of the present disclosure provides a method for generating point cloud data, including:

acquiring a scene image corresponding to the current scene collected by the target device, and the positioning pose information of the target device;

Extract at least one two-dimensional feature point from the scene image by using the feature point extraction method corresponding to the synchronous positioning and mapping system;

According to the positioning pose information and the position information of the two-dimensional feature points in the scene image, determine the three-dimensional feature points matching the two-dimensional feature points in the pre-built three-dimensional scene map, and the 3D position information of 3D feature points;

Based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, point cloud data corresponding to the current scene is generated.

Using the above method, the feature point extraction method corresponding to the SLAM system can be used to extract at least one two-dimensional feature point from the collected scene image, and according to the obtained positioning pose information and the position of the two-dimensional feature point in the scene image information, to determine the 3D feature points that match the 2D feature points in the 3D scene map, and the 3D position information of the 3D feature points. The three-dimensional feature points matched by the feature points and the three-dimensional position information of the three-dimensional feature points can further generate relatively accurate point cloud data corresponding to the current scene. Since the feature point extraction method corresponding to the SLAM system is used to extract the two-dimensional feature points in the scene image, the two-dimensional feature points are matched with the SLAM system, so that the generated point cloud data of the current scene is matched with the SLAM system, and the subsequent The accumulated error of the SLAM system is corrected by using the point cloud data corresponding to the current scene.

In a possible implementation, the method further includes:

determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

The generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, including:

Based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or the position confidence, the point cloud data corresponding to the current scene is generated.

In the above-mentioned embodiment, by determining the semantic information of the three-dimensional feature points and/or the position confidence corresponding to the three-dimensional position information of the three-dimensional feature points, based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or The position confidence is used to generate point cloud data corresponding to the current scene. The generated point cloud data contains rich information of 3D feature points.

In a possible implementation, the method further includes:

determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

determining a credible three-dimensional feature point in at least one three-dimensional feature point based on the semantic information and/or the position confidence of the three-dimensional feature point;

The generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, including:

Based on the trusted three-dimensional feature points and the three-dimensional position information corresponding to the trusted three-dimensional feature points, point cloud data corresponding to the current scene is generated.

Using the above method, the credible 3D feature point can be determined by the semantic information and/or the position confidence of the 3D feature point, and the unreliable 3D feature points in at least one 3D feature point can be screened out. 3D position information and credible 3D feature points can generate more accurate point cloud data corresponding to the current scene, which alleviates the bad influence of unreliable 3D feature points on point cloud data.

In a possible implementation manner, after generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, the method further includes:

Using the point cloud data corresponding to the current scene, the current positioning result of the SLAM system is adjusted to obtain the adjusted current positioning result.

Here, at least one two-dimensional feature point is extracted from the scene image by using the feature point extraction method corresponding to the SLAM system, that is, the type of the obtained two-dimensional feature point is the same as that of the feature point extracted by the SLAM system; for example, if the SLAM system The corresponding feature point extraction method is the FAST feature point extraction algorithm, then use the FAST feature point extraction algorithm to extract at least one two-dimensional feature point from the scene image, and the obtained two-dimensional feature point is the FAST corner point, and the SLAM system. The feature points are also FAST corner points, so the extracted two-dimensional feature points are of the same type as the feature points extracted in the SLAM system, so that the generated point cloud data corresponding to the current scene can be used to more accurately adjust the current SLAM system. Positioning results.

At the same time, compared with using the acquired pose data of the target device to eliminate the cumulative error of the SLAM system, the stability of the positioning results of the SLAM system can be improved.

In a possible implementation manner, acquiring the positioning pose information of the target device includes:

Determine the positioning pose information of the target device based on the scene image; or,

acquiring detection data of a positioning sensor included on the target device;

Based on the detection data, the positioning pose information of the target device is determined.

Here, setting up multiple ways to obtain the positioning pose information of the target device can improve the flexibility of determining the positioning pose information.

In a possible implementation manner, the obtaining of the scene image corresponding to the current scene collected by the target device and the positioning pose information of the target device include:

In the case of detecting that the target device satisfies the set movement conditions, acquire the scene image of the current scene collected by the target device, and the positioning pose information of the target device;

Wherein, that the target device satisfies the set movement condition includes: the movement distance of the target device reaches the set distance threshold; or, the movement time of the target device reaches the set time threshold.

The following descriptions of the effects of devices, electronic devices, etc. refer to the descriptions of the above-mentioned methods.

On the other hand, an embodiment of the present disclosure provides an apparatus for generating point cloud data, including:

an acquisition part, configured to acquire the scene image corresponding to the current scene collected by the target device, and the positioning pose information of the target device;

The extraction part is configured to extract at least one two-dimensional feature point from the scene image by using the feature point extraction method corresponding to the synchronous positioning and the mapping SLAM system;

A first determining part is configured to determine, according to the positioning pose information and the position information of the two-dimensional feature points in the scene image, matching with the two-dimensional feature points in a pre-built three-dimensional scene map The three-dimensional feature points of , and the three-dimensional position information of the three-dimensional feature points;

The generating part is configured to generate point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points.

In a possible implementation manner, the apparatus further includes: a second determining part configured to:

determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

The generation section is also configured to:

Based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or the position confidence, the point cloud data corresponding to the current scene is generated.

In a possible implementation manner, the apparatus further includes: a third determining part configured to:

determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

determining a credible three-dimensional feature point in at least one three-dimensional feature point based on the semantic information and/or the position confidence of the three-dimensional feature point;

The generation section is also configured to:

Based on the trusted three-dimensional feature points and the three-dimensional position information corresponding to the trusted three-dimensional feature points, point cloud data corresponding to the current scene is generated.

In a possible implementation manner, the device further includes: an adjustment part configured to:

Using the point cloud data corresponding to the current scene, the current positioning result of the SLAM system is adjusted to obtain the adjusted current positioning result.

In a possible implementation manner, the obtaining part is further configured as:

Determine the positioning pose information of the target device based on the scene image; or,

acquiring detection data of a positioning sensor included on the target device;

Based on the detection data, the positioning pose information of the target device is determined.

In a possible implementation manner, the obtaining part is further configured as:

In the case of detecting that the target device satisfies the set movement conditions, acquire the scene image of the current scene collected by the target device, and the positioning pose information of the target device;

Wherein, that the target device satisfies the set movement condition includes: the movement distance of the target device reaches the set distance threshold; or, the movement time of the target device reaches the set time threshold.

On the other hand, an embodiment of the present disclosure provides an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processor In communication with the memory through a bus, the machine-readable instructions execute the steps of the method for generating point cloud data according to the first aspect or any one of the embodiments when executed by the processor.

On the other hand, an embodiment of the present disclosure provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor when it is executed as described in the first aspect or any one of the above-mentioned embodiments. The steps of the point cloud data generation method.

On the other hand, an embodiment of the present disclosure provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the first aspect or any of the above The steps of the method for generating point cloud data according to an embodiment.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the embodiments of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

FIG. 1 shows a schematic flowchart of a method for generating point cloud data provided by an embodiment of the present disclosure;

FIG. 2 shows a schematic flowchart of a method for determining credible three-dimensional feature points in a method for generating point cloud data provided by an embodiment of the present disclosure;

FIG. 3 shows another method for generating point cloud data provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of the architecture of an apparatus for generating point cloud data provided by an embodiment of the present disclosure;

FIG. 5 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments These are only some of the embodiments of the present disclosure, but not all of the embodiments. The components of the disclosed embodiments generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

Real-time positioning and map construction means that a mobile device starts to move from an unknown position in an unknown environment, performs its own positioning according to the position estimation and map during the movement process, and builds an incremental map on the basis of its own positioning, so as to realize Autonomous positioning and navigation of mobile devices. Generally, a SLAM system running on a mobile device will generate a large error accumulation in long-distance tracking, which reduces the accuracy and stability of the SLAM system.

Since the offline maps generated by lidar or three-dimensional reconstruction (Struct From Motion, SFM) have high accuracy and global consistency, high-precision offline map point information can be integrated into the tracking process of the SLAM system to effectively Reduce the error of the SLAM system. Generally, the local image can be uploaded to the cloud for visual positioning, and the inliers can be screened out according to the positioning results of the current image and the offline map, and the inliers can be returned to the SLAM system. However, this method usually results in a limited number of inliers after screening, and it is difficult to continuously apply them to the SLAM system.

In order to solve the above problems, the embodiments of the present disclosure provide a method, apparatus, electronic device, and storage medium for generating point cloud data.

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all of the embodiments. The components of the disclosed embodiments generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present disclosure, but is merely representative of selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the embodiments of the present disclosure.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In order to facilitate the understanding of the embodiments of the present disclosure, a method for generating point cloud data disclosed in the embodiments of the present disclosure is first introduced in detail. The method for generating point cloud data provided by the embodiments of the present disclosure can be applied to a mobile computer device with a certain computing capability, for example, the mobile computer device can be a mobile phone, a computer, a tablet, an Augmented Reality (AR) device, Robots etc. In some possible implementations, the method for generating point cloud data may be implemented by a processor invoking computer-readable instructions stored in a memory.

Referring to FIG. 1, which is a schematic flowchart of a method for generating point cloud data provided by an embodiment of the present disclosure, the method includes S101-S104, wherein:

S101, acquiring a scene image corresponding to a current scene collected by a target device, and positioning pose information of the target device;

S102, utilize the feature point extraction mode corresponding to synchronous positioning and mapping system, extract at least one two-dimensional feature point from the scene image;

S103, according to the positioning pose information and the position information of the two-dimensional feature point in the scene image, determine the three-dimensional feature point matching the two-dimensional feature point in the pre-built three-dimensional scene map, and three-dimensional position information of the three-dimensional feature point;

S104 , based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, generate point cloud data corresponding to the current scene.

In the above method, the feature point extraction method corresponding to the SLAM system can be used to extract at least one two-dimensional feature point from the collected scene image, and according to the obtained positioning pose information and the position of the two-dimensional feature point in the scene image. information, to determine the 3D feature points that match the 2D feature points in the 3D scene map, and the 3D position information of the 3D feature points. The three-dimensional feature points matched by the feature points and the three-dimensional position information of the three-dimensional feature points can further generate relatively accurate point cloud data corresponding to the current scene. Since the feature point extraction method corresponding to the SLAM system is used to extract the two-dimensional feature points in the scene image, the two-dimensional feature points are matched with the SLAM system, so that the generated point cloud data of the current scene is matched with the SLAM system, and the subsequent The accumulated error of the SLAM system is corrected by using the point cloud data corresponding to the current scene.

S101-S104 will be described below.

For S101:

The target device may be any movable device including an image acquisition device, for example, the target device may be a robot, an AR device, a mobile phone, a computer, and the like. Here, the scene image of the current scene collected by the image collection device set on the target device may be acquired; the image collection device may be a camera or the like.

Here, a scene image corresponding to the current scene collected by the target device can be obtained, and the positioning pose information of the target device in the process of collecting the scene image can be obtained. The positioning pose information may include position information and orientation information, for example, the position information may be three-dimensional position information; the orientation information may be represented by Euler angles.

In an optional implementation manner, obtaining the positioning pose information of the target device in the above S101 may be implemented by at least one of the following implementation manners:

Manner 1: Determine the positioning pose information of the target device based on the scene image.

Manner 2: Acquire detection data of a positioning sensor included on the target device; and determine the positioning pose information of the target device based on the detection data.

In the first mode, a visual positioning algorithm may be used to determine the positioning pose information of the target device based on the scene image corresponding to the current scene. For example, feature point extraction can be performed on the scene image to obtain multiple feature point information included in the scene image, and the positioning pose information of the target device can be determined by using the multiple feature point information and the constructed offline map.

In the second manner, the positioning sensor may include: a radar device, an inertial measurement unit (Inertial Measurement Unit, IMU), a gyroscope and other sensors capable of measuring the position and attitude of the device. For example, when the positioning sensor is a radar device, the radar device can collect the point cloud data of the current scene, and then match the collected point cloud data with the high-precision map to determine the positioning pose information of the target device.

Wherein, the method for determining the positioning pose information of the target device may also include other positioning methods, which are only illustratively described herein. For example, the positioning pose information of the target device can also be determined by positioning methods such as a global positioning system (Global Positioning System, GPS), wireless communication technology WiFi positioning, and real-time dynamic positioning (Real-Timekinematic, RTK).

Here, setting up multiple ways to obtain the positioning pose information of the target device can improve the flexibility of determining the positioning pose information.

In an optional implementation manner, in S101, acquiring the scene image corresponding to the current scene collected by the target device and the positioning pose information of the target device may include:

In the case of detecting that the target device satisfies the set movement conditions, acquire the scene image of the current scene collected by the target device, and the positioning pose information of the target device;

Wherein, that the target device satisfies the set movement condition includes: the movement distance of the target device reaches the set distance threshold; or, the movement time of the target device reaches the set time threshold.

In an optional implementation manner, the scene corresponding to the current scene collected by the target device can be obtained when the moving distance of the target device reaches the set distance threshold, or when the moving time of the target device reaches the time threshold of the device. The image and the positioning pose information of the target device. The distance threshold and time threshold can be set as required, for example, the distance threshold can be 20 meters, 30 meters, 50 meters, etc.; the time threshold can be 30 seconds, 1 minute, and so on.

For example, the scene image of the current scene collected by the target device and the positioning pose information of the target device may be acquired once every time the target device moves 20 meters (distance threshold). Alternatively, when the target device moves every 20 seconds (time threshold), a scene image of the current scene collected by the target device and the positioning pose information of the target device are acquired once.

Exemplarily, the movement distance of the target device may be determined by using a displacement sensor provided on the target device for measuring the movement distance. Alternatively, the set positioning algorithm can be used to perform real-time detection on the moving distance of the target device, etc. The moving time of the target device can be determined using the clock set on the target device.

For S102 and S103:

At least one two-dimensional feature point can be extracted from the acquired scene image of the current scene by using the feature point extraction method corresponding to the SLAM system. For example, the two-dimensional feature points may be feature points on the target object included in the scene image.

Here, at least one two-dimensional feature point can be extracted from the scene image corresponding to the current scene by using the feature point extraction method corresponding to the SLAM system. The feature point extraction method may be a feature point extraction algorithm deployed in the SLAM system. For example, the feature point extraction algorithm may include, but is not limited to, Scale Invariant Feature Transform (SIFT) algorithm, Scale Invariant Feature Transform The accelerated version of the algorithm (SIFT algorithm) SURF algorithm, FAST feature point extraction algorithm, etc.

For example, if the feature point extraction algorithm corresponding to the SLAM system is the FAST feature point extraction algorithm, the FAST feature point extraction algorithm may be used to extract at least one two-dimensional feature point from the scene image.

Here, the step of extracting at least one two-dimensional feature point from the scene image by using the feature point extraction algorithm corresponding to the SLAM system can be executed on the mobile device; the feature point extraction algorithm corresponding to the SLAM system can also be executed on the server, from the scene image. The step of extracting at least one two-dimensional feature point from the image.

For example, after the target device collects the scene image, at least one two-dimensional feature point can be extracted from the scene image by using the feature point extraction algorithm corresponding to the SLAM system set on the movable device. Alternatively, after the target device collects the scene image, the collected scene image can be sent to the server, so that at least one two-dimensional feature point can be extracted from the scene image by using the feature point extraction algorithm corresponding to the SLAM system set on the server.

After the 2D feature points are obtained, the 3D feature points matching the 2D feature points and the 3D feature points can be determined in the pre-built 3D scene map according to the positioning pose information and the position information of the 2D feature points in the scene image. The three-dimensional position information corresponding to the feature point.

Exemplarily, a ray casting algorithm can be used to determine the three-dimensional feature points that match the two-dimensional feature points according to the positioning pose information, the position information of the two-dimensional feature points in the scene image, and the pre-built three-dimensional scene map. , and the three-dimensional position information corresponding to the three-dimensional feature points.

For S104:

The point cloud data corresponding to the current scene can be generated by using the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points.

In a possible implementation manner, the method further includes: determining semantic information of the three-dimensional feature point, and/or position confidence corresponding to the three-dimensional position information of the three-dimensional feature point.

The generating the point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points includes: based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or the position confidence, to generate point cloud data corresponding to the current scene.

In the above-mentioned embodiment, by determining the semantic information of the three-dimensional feature points and/or the position confidence corresponding to the three-dimensional position information of the three-dimensional feature points, based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or The position confidence is used to generate point cloud data corresponding to the current scene. The generated point cloud data contains rich information of 3D feature points.

Exemplarily, in the pre-built three-dimensional scene map, the information corresponding to each three-dimensional feature point may include semantic information and/or position confidence information. The semantic information corresponding to the three-dimensional feature point and/or the position confidence corresponding to the three-dimensional position information of the three-dimensional feature point are obtained from the three-dimensional scene map. The position confidence can be used to represent the reliability of the three-dimensional position information.

In a possible implementation, in the process of constructing the three-dimensional scene map, the semantic information and position confidence of the three-dimensional feature points may be determined in the pre-built three-dimensional scene map; In the process of information or position confidence, the semantic information and position confidence of the three-dimensional feature points in the pre-built three-dimensional scene map are determined.

The three-dimensional scene map can be constructed according to the following steps: obtaining the video corresponding to the scene, sampling the multi-frame scene samples from the video, or obtaining the collected multi-frame scene samples corresponding to the scene; A plurality of three-dimensional feature point information is extracted from the sample; and a three-dimensional scene map can be constructed based on the extracted multiple three-dimensional feature point information.

In the process that the 3D feature points in the 3D scene map contain semantic information, the trained semantic segmentation neural network can be used to detect the constructed 3D scene map, and determine the semantic information of each 3D feature point in the 3D scene map. The semantic information of the three-dimensional feature point may be used to represent the type of the target object corresponding to the three-dimensional feature point. For example, the semantic information of the three-dimensional feature point may include walls, tables, cups, leaves, animals, and the like. Here, the semantic information of the three-dimensional feature points can be set as required.

In the process that the 3D feature points in the 3D scene map contain the position confidence, the trained neural network can be used to detect the constructed 3D scene map to determine the position confidence of each 3D feature point in the 3D scene map. Alternatively, the position confidence of each 3D feature point in the 3D scene map can also be determined according to the semantic information of the 3D feature point. For example, if the semantic information of the 3D feature point is a table, since the table is an object that is not easy to move, the confidence of the position of the 3D feature point can be set higher; if the semantic information of the 3D feature point is a leaf, because the leaf is relatively The object is easy to move, so the position confidence of the three-dimensional feature point can be set to be small.

Further, point cloud data corresponding to the current scene may be generated based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or position confidence. For example, in the process that the three-dimensional feature points include semantic information, the generated point cloud data corresponding to the current scene includes the semantic information of each point cloud point.

In a possible implementation, as shown in FIG. 2 , the method further includes:

S201, determining the semantic information of the three-dimensional feature point, and/or, the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

S202: Determine a credible three-dimensional feature point in at least one three-dimensional feature point based on the semantic information and/or the position confidence of the three-dimensional feature point.

The generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points includes: based on the trusted three-dimensional feature points and the trusted three-dimensional feature points The corresponding three-dimensional position information is used to generate point cloud data corresponding to the current scene.

Using the above method, the credible 3D feature point can be determined by the semantic information and/or the position confidence of the 3D feature point, and the unreliable 3D feature points in at least one 3D feature point can be screened out. 3D position information and credible 3D feature points can generate more accurate point cloud data corresponding to the current scene, which alleviates the bad influence of unreliable 3D feature points on point cloud data.

For the manner of determining the position confidence corresponding to the semantic information of the three-dimensional feature point and the three-dimensional position information of the three-dimensional feature point, reference may be made to the above process.

The trusted three-dimensional feature points may be determined based on semantic information and/or position confidence of the three-dimensional feature points.

In the process of determining a credible 3D feature point in at least one 3D feature point based on the semantic information of the 3D feature point, it can be determined whether the object corresponding to the 3D feature point belongs to the movable category according to the semantic information of the 3D feature point, and if so, It is determined that the three-dimensional feature point does not belong to the credible three-dimensional feature point; if not, it is determined that the three-dimensional feature point belongs to the credible three-dimensional feature point. Among them, a mapping relationship table of movable categories and immovable categories can be preset, and then it can be determined according to the semantic information of the three-dimensional feature points and the set mapping relationship table that the objects corresponding to the three-dimensional feature points belong to the movable category or belong to the immovable category. category.

In the process of determining a credible 3D feature point in at least one 3D feature point based on the position confidence of the 3D feature points, a confidence threshold may be set, and a 3D feature point whose position confidence is greater than or equal to the confidence threshold may be determined as Credible 3D feature points; 3D feature points whose position confidence is less than the set confidence threshold are determined as unreliable 3D feature points.

In the process of determining a credible 3D feature point in at least one 3D feature point based on the semantic information and position confidence of the 3D feature point, the candidate can be determined based on the semantic information of the 3D feature point first in the at least one 3D feature point. The trusted three-dimensional feature points are determined; and the trusted three-dimensional feature points in the candidate trusted three-dimensional feature points are determined based on the position confidence. Alternatively, a candidate credible 3D feature point in at least one 3D feature point may be determined based on the position confidence of the 3D feature point; then based on semantic information, a credible 3D feature point in the candidate credible 3D feature point may be determined.

Furthermore, point cloud data corresponding to the current scene may be generated based on the trusted three-dimensional feature points and the three-dimensional position information corresponding to the trusted three-dimensional feature points.

In a possible implementation, based on the method provided in FIG. 1 , after generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, the method further includes S301 , see FIG. 3 . shown:

S301 , using the point cloud data corresponding to the current scene, adjust the current positioning result of the SLAM system to obtain an adjusted current positioning result.

Here, you can input the point cloud data corresponding to the current scene into the SLAM system, control the SLAM system to add the received point cloud data to the tracking process of the SLAM system, and adjust the current positioning results of the SLAM system to eliminate SLAM The errors of the system are accumulated to obtain the adjusted current positioning result, so that the obtained adjusted current positioning result has a higher accuracy.

Here, at least one two-dimensional feature point is extracted from the scene image by using the feature point extraction method corresponding to the SLAM system, that is, the type of the obtained two-dimensional feature point is the same as that of the feature point extracted by the SLAM system; for example, if the SLAM system The corresponding feature point extraction method is the FAST feature point extraction algorithm, then use the FAST feature point extraction algorithm to extract at least one two-dimensional feature point from the scene image, and the obtained two-dimensional feature point is the FAST corner point, and the SLAM system. The feature points are also FAST corner points, so the extracted two-dimensional feature points are of the same type as the feature points extracted in the SLAM system, so that the generated point cloud data corresponding to the current scene can be used to more accurately adjust the current SLAM system. Positioning results. At the same time, compared with using the acquired pose data of the target device to eliminate the cumulative error of the SLAM system, the stability of the positioning results of the SLAM system can be improved.

Those skilled in the art can understand that, in the above-mentioned method of the specific embodiment, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the execution order of each step should be based on its function and possible intrinsic Logical OK.

Based on the same concept, an embodiment of the present disclosure also provides an apparatus for generating point cloud data. Referring to FIG. 4 , a schematic diagram of the architecture of the apparatus for generating point cloud data provided by an embodiment of the present disclosure includes an acquisition part 401 and an extraction part 402 , a first determining part 403, and a generating part 404, wherein:

The acquisition part 401 is configured to acquire the scene image corresponding to the current scene collected by the target device, and the positioning pose information of the target device;

The extraction part 402 is configured to extract at least one two-dimensional feature point from the scene image by using a feature point extraction method corresponding to the synchronous positioning and mapping SLAM system;

The first determining part 403 is configured to determine, according to the positioning pose information and the position information of the two-dimensional feature points in the scene image, the pre-built three-dimensional scene map, and the two-dimensional feature points and the two-dimensional feature points. matching three-dimensional feature points and three-dimensional position information of the three-dimensional feature points;

The generating part 404 is configured to generate point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points.

In a possible implementation manner, the apparatus further includes: a second determining part 405 configured to:

determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

The generating section 404 is further configured to:

Based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or the position confidence, the point cloud data corresponding to the current scene is generated.

In a possible implementation manner, the apparatus further includes: a third determining part 406 configured to:

determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point;

determining a credible three-dimensional feature point in at least one three-dimensional feature point based on the semantic information and/or the position confidence of the three-dimensional feature point;

The generating section 404 is further configured to:

Based on the trusted three-dimensional feature points and the three-dimensional position information corresponding to the trusted three-dimensional feature points, point cloud data corresponding to the current scene is generated.

In a possible implementation manner, the device further includes: an adjustment part 407 configured to:

Using the point cloud data corresponding to the current scene, the current positioning result of the SLAM system is adjusted to obtain the adjusted current positioning result.

In a possible implementation manner, the obtaining part 401 is further configured to:

Determine the positioning pose information of the target device based on the scene image; or,

Acquire detection data of a positioning sensor included on the target device; and determine the positioning pose information of the target device based on the detection data.

In a possible implementation manner, the obtaining part 401 is further configured to:

In the case of detecting that the target device satisfies the set movement conditions, acquire the scene image of the current scene collected by the target device, and the positioning pose information of the target device;

Wherein, that the target device satisfies the set movement condition includes: the movement distance of the target device reaches the set distance threshold; or, the movement time of the target device reaches the set time threshold.

In some embodiments, the functions or templates included in the apparatus provided by the embodiments of the present disclosure may be configured to execute the methods described in the above method embodiments, and for implementation, reference may be made to the above method embodiments.

Based on the same technical concept, an embodiment of the present disclosure also provides an electronic device. Referring to FIG. 4 , a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure includes a processor 401 , a memory 402 , and a bus 403 . Among them, the memory 402 is configured to store execution instructions, including the memory 4021 and the external memory 4022; the memory 4021 here is also called internal memory, and is configured to temporarily store the operation data in the processor 401 and the external memory 4022 such as the hard disk. Data, the processor 401 exchanges data with the external memory 4022 through the memory 4021. When the electronic device 400 is running, the processor 401 and the memory 402 communicate through the bus 403, so that the processor 401 executes the following instructions:

acquiring a scene image corresponding to the current scene collected by the target device, and the positioning pose information of the target device;

Extract at least one two-dimensional feature point from the scene image by using the feature point extraction method corresponding to the synchronous positioning and mapping SLAM system;

According to the positioning pose information and the position information of the two-dimensional feature points in the scene image, determine the three-dimensional feature points matching the two-dimensional feature points in the pre-built three-dimensional scene map, and the 3D position information of 3D feature points;

Based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, point cloud data corresponding to the current scene is generated.

In addition, embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the method for generating point cloud data described in the foregoing method embodiments is executed A step of.

Embodiments of the present disclosure further provide a computer program product, where the computer program product carries program codes, and the instructions included in the program codes can be used to execute the steps of the point cloud data generation method described in the foregoing method embodiments. Method Examples.

Wherein, the above-mentioned computer program product can be realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, reference may be made to the corresponding processes in the foregoing method embodiments for the working processes of the systems and apparatuses described above. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present disclosure are essentially or contribute to the prior art or parts of the technical solutions may be embodied in the form of software products, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure.

In some possible implementations, the computer-readable storage medium may be a ferroelectric memory (Ferroelectric Random Access Memory, FRAM), a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable read-only memory) , PROM), Electronic Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash Memory, Magnetic Surface Memory, Optical Disc, or CD - ROM and other memories; can also be various devices including one or any combination of the above memories.

The above are only implementations of the embodiments of the present disclosure, but the protection scope of the embodiments of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the embodiments of the present disclosure. , all should be covered within the protection scope of the embodiments of the present disclosure. Therefore, the protection scope of the embodiments of the present disclosure should be subject to the protection scope of the claims.

Industrial Applicability

The present disclosure provides a point cloud data generation method, device, electronic device and storage medium. The method includes: acquiring a scene image corresponding to a current scene collected by a target device, and positioning pose information of the target device; using synchronous positioning The feature point extraction method corresponding to the mapping SLAM system, extracts at least one two-dimensional feature point from the scene image; according to the positioning pose information and the position information of the two-dimensional feature point in the scene image , determine the three-dimensional feature points that match the two-dimensional feature points and the three-dimensional position information of the three-dimensional feature points in the pre-built three-dimensional scene map; based on the three-dimensional feature points and the three-dimensional position of the three-dimensional feature points information to generate point cloud data corresponding to the current scene. Since the feature point information included in the three-dimensional scene map is relatively accurate, the embodiment of the present disclosure can more accurately determine the three-dimensional feature points that match the two-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, and further, can generate the corresponding more accurate point cloud data. Since the feature point extraction method corresponding to the SLAM system is used to extract the two-dimensional feature points in the scene image, the two-dimensional feature points are matched with the SLAM system, so that the generated point cloud data of the current scene is matched with the SLAM system, and the subsequent The accumulated error of the SLAM system is corrected by using the point cloud data corresponding to the current scene.

Claims (11)

A method for generating point cloud data, comprising: acquiring a scene image corresponding to the current scene collected by the target device, and the positioning pose information of the target device; Extract at least one two-dimensional feature point from the scene image by using the feature point extraction method corresponding to the synchronous positioning and mapping SLAM system; According to the positioning pose information and the position information of the two-dimensional feature points in the scene image, determine the three-dimensional feature points matching the two-dimensional feature points and the three-dimensional feature points in the pre-built three-dimensional scene map. Three-dimensional position information of feature points; Based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, point cloud data corresponding to the current scene is generated. The method of claim 1, wherein the method further comprises: determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point; The generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, including: Based on the three-dimensional feature points, the three-dimensional position information of the three-dimensional feature points, and the determined semantic information and/or the position confidence, the point cloud data corresponding to the current scene is generated. The method of claim 1, wherein the method further comprises: determining the semantic information of the three-dimensional feature point, and/or the position confidence level corresponding to the three-dimensional position information of the three-dimensional feature point; determining a credible three-dimensional feature point in the at least one three-dimensional feature point based on the semantic information and/or the position confidence of the three-dimensional feature point; The generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, including: Based on the trusted three-dimensional feature points and the three-dimensional position information corresponding to the trusted three-dimensional feature points, point cloud data corresponding to the current scene is generated. The method according to any one of claims 1 to 3, wherein after generating point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points, the method Also includes: Using the point cloud data corresponding to the current scene, the current positioning result of the SLAM system is adjusted to obtain the adjusted current positioning result. The method according to claim 1, wherein the acquiring the positioning pose information of the target device comprises: Determine the positioning pose information of the target device based on the scene image; or, Acquire detection data of a positioning sensor included on the target device, and determine the positioning pose information of the target device based on the detection data. The method according to any one of claims 1 to 5, wherein the obtaining the scene image corresponding to the current scene collected by the target device and the positioning pose information of the target device includes: Obtaining a scene image corresponding to the current scene collected by the target device and the positioning pose information of the target device when it is detected that the target device satisfies the set movement conditions; Wherein, the target device satisfies the set movement conditions including at least one of the following: the movement distance of the target device reaches the set distance threshold; the movement time of the target device reaches the set time threshold. A device for generating point cloud data, comprising: an acquisition part, configured to acquire the scene image corresponding to the current scene collected by the target device, and the positioning pose information of the target device; The extraction part is configured to extract at least one two-dimensional feature point from the scene image by using the feature point extraction method corresponding to the synchronous positioning and the mapping SLAM system; A first determining part is configured to determine, according to the positioning pose information and the position information of the two-dimensional feature points in the scene image, matching with the two-dimensional feature points in a pre-built three-dimensional scene map The three-dimensional feature points of , and the three-dimensional position information of the three-dimensional feature points; The generating part is configured to generate point cloud data corresponding to the current scene based on the three-dimensional feature points and the three-dimensional position information of the three-dimensional feature points. The apparatus according to claim 7, wherein the acquisition part is further configured to: Determine the positioning pose information of the target device based on the scene image; or, Acquire detection data of a positioning sensor included on the target device, and determine the positioning pose information of the target device based on the detection data. An electronic device, comprising: a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus , the machine-readable instructions execute the steps of the point cloud data generation method according to any one of claims 1 to 6 when the machine-readable instructions are executed by the processor. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for generating point cloud data according to any one of claims 1 to 6 are executed . A computer program, comprising computer readable code, when the computer readable code is executed in an electronic device, the point cloud of any one of claims 1 to 6 is realized when executed by a processor in the electronic device The steps of the data generation method.

Images