WO2025242011A1 - Three-dimensional model reconstruction method based on multi-source input and apparatus - Google Patents

Abstract

The present application provides a three-dimensional model reconstruction method based on multi-source input, which is applied to a three-dimensional model reconstruction platform, and comprises: acquiring a mapping result comprising image information and point cloud information of an object to be reconstructed; establishing a three-dimensional model; from the mapping result, determining target image information and target point cloud information of said object under observation at a target pose; instructing the three-dimensional model to generate a target depth image and a target digital image on the basis of the target pose; then separately determining a corresponding geometric residual on the basis of the target point cloud information and the target depth image, and determining a corresponding optical residual on the basis of the target image information and the target digital image; and adjusting the three-dimensional model on the basis of the geometric residual and the optical residual. Therefore, the efficiency and precision of three-dimensional model reconstruction are improved, allowing rendered products of obtained three-dimensional models to be more realistic.

Description

A method and apparatus for three-dimensional model reconstruction based on multi-source input

This disclosure claims priority to Chinese Patent Application No. 202410626989.0, filed on May 20, 2024, entitled "A Three-Dimensional Reconstruction Method and Apparatus Based on Cloud Computing Technology", and Chinese Patent Application No. 202411096682.0, filed on August 9, 2024, entitled "A Three-Dimensional Model Reconstruction Method and Apparatus Based on Multi-Source Input", the entire contents of which are incorporated herein by reference.

Technical Field

This application relates to the field of three-dimensional model reconstruction, and in particular to a method and apparatus for three-dimensional model reconstruction based on multi-source input.

Background Technology

3D reconstruction is a mathematical process and computer technology for restoring the original three-dimensional information of a scene. Early 3D model reconstruction methods typically used single-view or multi-view image information as input. Limited by the input data, the reconstructed 3D models were usually incomplete and lacked realism. High-precision 3D reconstruction schemes based on pure visual input had cumbersome data acquisition processes and low work efficiency. Due to the heavy reliance on the Structure from Motion (SFM) module, the applicable scenarios were limited, the degree of automation was low, it was difficult to meet the requirements of high speed, batch processing, and high frequency, and the accuracy and precision were low with severe distortion.

In recent years, with the iteration of sensor hardware technology, reconstruction technology that integrates three-dimensional sensor information such as LiDAR (Light Detection and Range Sensor) and Inertial Measurement Unit (IMU) has developed rapidly, aiming to obtain denser and more accurate three-dimensional models.

Summary of the Invention

This application provides a method and apparatus for reconstructing a three-dimensional model based on multi-source input. By acquiring mapping results including point cloud information and image information of the object to be reconstructed, a three-dimensional model is established. Furthermore, geometric residuals and optical residuals are determined through point cloud information and image information, and the three-dimensional model is further adjusted to improve modeling efficiency, modeling accuracy, and modeling realism.

Firstly, this application provides a 3D model reconstruction method based on multi-source input. This method is applied to a 3D model reconstruction platform. Specifically, the method includes the following steps: obtaining the mapping results of the object to be reconstructed, wherein the mapping results include image information and point cloud information of the object to be reconstructed. Further, a 3D model of the object to be reconstructed is established based on the mapping results. On this basis, target image information and target point cloud information of the object to be reconstructed under target pose observation are determined from the mapping results. The target pose includes position information and attitude information. This instructs the 3D model to generate a target depth image and a target digital image based on the target pose. Thus, the 3D model reconstruction platform determines the geometric residual based on the target point cloud information and the target depth image, determines the optical residual based on the target image information and the target digital image, and then adjusts the 3D model based on the geometric and optical residuals.

In the solution provided in this application, a 3D model is built by acquiring mapping results from multiple inputs, including point cloud information and image information of the object to be reconstructed. This allows the built 3D model to output corresponding depth and digital images based on specific poses. Then, by combining the target point cloud information and target image information from the mapping results, optical and geometric residuals are constructed. The 3D model is then adjusted based on these optical and geometric residuals, jointly supervising the training of the 3D model. Thus, by simultaneously employing both geometric and optical supervision to adjust the 3D model to possess realistic optical and geometric information, the geometric accuracy and detail feature richness of the 3D model are significantly improved, resulting in highly realistic rendered objects.

In conjunction with the first aspect, in one possible implementation of the first aspect, the specific steps for obtaining the mapping result of the object to be reconstructed are as follows: obtain the initial image information and initial point cloud information of the object to be reconstructed, as well as the image inertial information and point cloud inertial information corresponding to the acquisition device when acquiring the initial image information and the acquisition device when acquiring the initial point cloud information, and then generate the mapping result based on the image information, point cloud information, point cloud inertial information and image inertial information, thereby obtaining the mapping result of the modeling object.

The solution provided in this application acquires multi-source input information such as image information, point cloud information, and corresponding inertial information, and fuses frame-by-frame point cloud, frame-by-frame image, and corresponding motion prior information to construct a map of the object to be reconstructed, achieving high-precision mapping. Furthermore, by introducing corresponding inertial information and other motion prior information, along with point cloud information, pose fitting can be performed more quickly, improving mapping efficiency.

In conjunction with the first aspect, in one possible implementation of the first aspect, the 3D model reconstruction platform determines descriptive information based on the initial image information, wherein the descriptive information includes the texture features of at least one frame of the initial image information, and then determines the keyframe image based on the initial image information and the descriptive information.

In the solution provided in this application, the determination of keyframe images helps to filter out images that meet the requirements of specific descriptor information. For example, some images with relatively rich texture features or details can be used to further improve the adjustment efficiency of the 3D model and the rendering effect of the 3D model on the details by selecting keyframe images to construct optical residuals when adjusting the 3D model.

In conjunction with the first aspect, in one possible implementation of the first aspect, the 3D model reconstruction platform can determine the target pose for observing the keyframe image based on the keyframe image.

In the solution provided in this application, the target pose is determined based on keyframe images, enabling the construction of optical residuals between the digital image output by the 3D model and the keyframe images with relatively rich texture features or detail. Simultaneously, it also enables the construction of geometric residuals between the depth image output by the 3D model and the corresponding point cloud information with high accuracy. Therefore, using the pose obtained from the keyframe images as the basis for residual construction can further improve the efficiency and effectiveness of 3D model training iterations.

In conjunction with the first aspect, in one possible implementation, the 3D model reconstruction platform can further determine image pose information based on image information and image inertial information, and determine point cloud pose information based on point cloud information and point cloud inertial information. Thus, a point cloud map is generated based on the image information, point cloud information, point cloud inertial information, and image inertial information. Specifically, this includes the following steps: generating a point cloud map based on image information, point cloud information, point cloud pose information, and image pose information. Furthermore, the 3D model reconstruction platform can determine global pose information based on the point cloud pose information and image pose information.

The solution provided in this application fuses image information with image inertial information, point cloud information, and point cloud inertial information to obtain corresponding image poses and point cloud poses. This allows for accurate mapping during image and point cloud construction, as the adjacent and matching relationships between the image and point cloud information can be determined based on their respective poses. By establishing global pose information, rapid matching and conversion between image and point cloud poses can be achieved, facilitating the output of a complete and accurate rendered image from the 3D model based on a specific pose from either the point cloud or the image.

In conjunction with the first aspect, in one possible implementation of the first aspect, the 3D model reconstruction platform performs loop closure detection on the mapping results to obtain matching information. The matching information is used to indicate the matching status between the current frame and historical frames, and the mapping results are adjusted according to the matching information.

In the solution provided in this application, due to the cumulative measurement errors of the acquisition device during long-term operation, especially when the acquisition device returns to a mapped area after a long period of operation, a misalignment may occur between the current sensor information and the already constructed mapping results. Therefore, loop closure detection is required during the mapping process to check whether the current frame can form a loop with historical information, construct a pose graph, adjust the mapping results, and improve the accuracy of the mapping.

In conjunction with the first aspect, in one possible implementation of the first aspect, the 3D model reconstruction platform acquires supplementary image information, supplementary point cloud information, supplementary image inertial information, and supplementary laser inertial information of the object to be reconstructed, and updates the mapping results based on the supplementary image information, supplementary point cloud information, supplementary laser inertial information, and supplementary image inertial information.

In the solution provided in this application, by adopting multi-source input for mapping the object to be reconstructed, the mapping speed is improved, and the current mapping progress can be fed back to the user in real time. This allows the user to know in a timely manner whether there are any local areas that have been missed or details that are missing within the object to be reconstructed during the mapping process, and to perform targeted supplementary scanning of the corresponding areas, thereby completing the construction of high-quality raw data for mapping.

In conjunction with the first aspect, in one possible implementation of the first aspect, the user can configure the adjustment of the 3D model. The 3D model reconstruction platform obtains the number of iterations input by the user and instructs the 3D model to be adjusted according to the number of iterations. The number of iterations is used to indicate the number of adjustments to the 3D model. After the 3D model meets the number of iterations, the 3D model is output.

In the solution provided in this application, the user controls the iterative training process of the 3D model through configuration, so that after the 3D model meets the corresponding adjustment requirements, the adjusted 3D model is output for subsequent service use.

The second aspect or any implementation thereof is a device implementation corresponding to the first aspect or any implementation thereof. The description in the first aspect or any implementation thereof applies to the second aspect or any implementation thereof, and will not be repeated here.

Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect and any implementation thereof.

Fourthly, this application provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in the first aspect and any implementation thereof.

Fifthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the method described in the first aspect and any implementation thereof.

Attached Figure Description

Figure 1 is a schematic diagram of an application scenario of the three-dimensional model reconstruction method based on multi-source input provided in the embodiments of this application;

Figure 2 is a schematic diagram of another application scenario of the three-dimensional model reconstruction method based on multi-source input provided in the embodiments of this application;

Figure 3 is a flowchart illustrating a three-dimensional model reconstruction method based on multi-source input provided in an embodiment of this application;

Figure 4 is a schematic diagram of a mapping process for a three-dimensional model reconstruction method based on multi-source input provided in an embodiment of this application;

Figure 5 is a schematic diagram of another mapping process of the three-dimensional model reconstruction method based on multi-source input provided in the embodiments of this application;

Figure 6 is a schematic diagram of a keyframe determination process for a three-dimensional model reconstruction method based on multi-source input provided in an embodiment of this application;

Figure 7 is a schematic diagram of a three-dimensional model reconstruction process based on a multi-source input three-dimensional model reconstruction method provided in an embodiment of this application;

Figure 8 is a schematic diagram of the device provided in an embodiment of this application;

Figure 9 is a schematic diagram of the computing device for the three-dimensional model reconstruction method based on multi-source input provided in the embodiments of this application;

Figure 10 is a schematic diagram of a computing device cluster for a three-dimensional model reconstruction method based on multi-source input provided in an embodiment of this application;

Figure 11 is another schematic diagram of the computing device cluster of the three-dimensional model reconstruction method based on multi-source input provided in the embodiments of this application;

Figure 12 is another schematic diagram of the computing device cluster of the three-dimensional model reconstruction method based on multi-source input provided in the embodiments of this application.

Detailed Implementation

The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

First, the terms appearing in this article will be explained as follows:

3D model reconstruction refers to the process of converting real-world objects or scenes into computer-processable 3D digital models using computer vision and graphics processing techniques.

Object to be reconstructed: This refers to the object or entity to be reconstructed using computer vision and graphics processing techniques. It can be a 3D space, a scene, an object-level model, or a digital human, etc. Depth Image: Also known as a distance image, a depth image is an image that uses the distance (depth) from the image acquisition device to various points in the scene as pixel values. It directly reflects the geometry of the visible surfaces of the scene. Depth images can be converted into point cloud data through coordinate transformation, and point cloud data with regularity and necessary information can also be converted back into depth image data.

A digital image is an image represented by a finite number of digital pixels, represented by an array or matrix, where the illumination position and intensity are discrete. A digital image is an image obtained by digitizing an analog image, using pixels as its basic element, and can be stored and processed by a digital computer or digital circuit. In one embodiment of this application, the digital image can be a binary image, a color image, etc., such as an RGB image or a panchromatic image.

Point clouds are datasets of points in space that can represent three-dimensional shapes or objects, typically acquired by a 3D scanner. For example, the position of each point in a point cloud is described by a set of Cartesian coordinates (X,Y,Z), and some may contain color information (R,G,B) or information about the intensity of the object's reflective surface. In one embodiment of this application, the point cloud information can be a point cloud directly acquired by a laser scanner (LiDAR) or other sensors; it can also be a point cloud directly acquired by other sensors, such as acoustic radar.

A 3D model is a mathematical representation of a real or fictional object in three-dimensional space. This representation typically consists of a series of 3D coordinate points connected to form lines and surfaces, creating a complete geometric structure that accurately reproduces the shape, structure, and appearance of the object, enabling scene reconstruction. Examples include 3D Gaussian Splatting (3DGS), a scene reconstruction and rendering technique based on 3D Gaussian volumes. It combines the advantages of explicit and implicit representations, enabling scene reconstruction and efficient real-time rendering based on pure image input, and generating synthetic data from new perspectives. It represents a new paradigm for reconstruction in the current field of computer vision. Another example is Neural Radiance Field (NeRF), an implicit 3D model reconstruction method based on deep neural networks. By learning the radiation and color information of each point in the scene, it can synthesize realistic images from any perspective. It constructs an implicit representation of the scene by sampling points in 3D space and predicting the radiation and color for each point.

Rasterization: The core step in 3D Gaussian sputtering rendering, it's a mathematical process that converts the mathematical description of an object and its associated color information into pixels on the screen for corresponding locations and the colors used to fill those pixels. In 3D Gaussian, it refers to the mathematical process of rendering a 2D image from the distribution and color information of the Gaussian kernel.

An inertial measurement unit (IMU) is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object.

Pose: refers to the position and orientation of an object relative to a reference coordinate system. Specifically, pose includes the object's spatial position information and its rotation direction information. Pose can be used to describe the position and orientation of any rigid body object in three-dimensional space. Position typically refers to the coordinates of the object's center or a specific reference point in three-dimensional space; orientation typically refers to the object's rotation angle or rotation matrix, representing the object's rotation relative to the reference coordinate system.

Explicit representation: The traditional form of representation for 3D models, which involves explicitly modeling a scene or object, allowing users to edit and view it, including meshes, point clouds, voxels, etc.

Implicit representation: Based on machine learning methods such as deep neural networks, it describes the representation of the three-dimensional information of the scene in a parameterized way, and constructs a mapping relationship from three-dimensional spatial coordinates to corresponding geometric/texture information.

Simultaneous Localization and Mapping (SLAM): A real-time explicit 3D model reconstruction method that aims to enable a robot to start from an unknown location in an unknown environment, locate its own position and orientation by repeatedly observing map features (such as corners, pillars, etc.) during movement, and then incrementally build a map based on its own position, thereby achieving the goal of simultaneous localization and map construction.

Structure from Motion (SFM): A non-real-time explicit 3D model reconstruction method that, given a series of images with a certain degree of overlap, simultaneously estimates the position and pose of the camera when each image was taken, as well as the sparse point cloud of the object or scene being photographed.

In the field of 3D reconstruction, using single-view or multi-view image information as input data to reconstruct the 3D model of the object to be reconstructed is a common method. However, due to limitations in input data, accurate point cloud information is often estimated using view images. View fusion is time-consuming, inefficient, and has poor accuracy. Furthermore, the reconstructed 3D model is incomplete, with severe distortion of depth information and a lack of realism. Moreover, the lack of accurate geometric verification information during the optimization process makes it difficult to optimize and adjust geometric depth information, further contributing to the low efficiency of 3D model reconstruction.

Based on this, this application provides a three-dimensional model reconstruction method based on multi-source input. By acquiring mapping results including point cloud information and image information of the object to be modeled and reconstructed, a three-dimensional model is established. Furthermore, geometric residuals and optical residuals are determined through point cloud information and image information, and the three-dimensional model is further adjusted to improve modeling efficiency, modeling accuracy, and modeling realism.

Please refer to Figure 1. As shown in Figure 1, Figure 1 is a schematic diagram of an application scenario of the 3D model reconstruction method based on multi-source input provided in the embodiments of this application. Specifically:

As one embodiment of this application, the 3D model reconstruction platform 12 can run on infrastructure 15, which includes at least one computing device for computation during the 3D model reconstruction process. User A can directly access the 3D model reconstruction platform 12, control the 3D model reconstruction process through it, and input corresponding operation commands during the process. The 3D model reconstruction platform 12 obtains the mapping result 16 of the object to be reconstructed and reconstructs a 3D model 17 based on the mapping result 16. The 3D model 17, as the output, can be used by subsequent data asset 3D model reconstruction platforms and digital twin simulation platforms. The digital asset 3D model reconstruction platform is a digital asset platform that can uniformly manage and schedule the reconstructed 3D model and various types of multimodal synthetic data derived from the 3D model. According to the digital asset format, it can be divided into original 3D models, color images, depth maps, color point clouds, semantic maps, individual segmentation results, and mesh models, etc. Synthetic data can be accumulated and can also be directly applied to the training of large-scale embodied models. In addition, the digital twin simulation platform can directly load dense 3D models and display highly realistic rendering results of 3D space to users in real time within the simulator interface. It supports users to roam in the simulation scene in real time and can also use the precise geometric information of the 3D model to realize downstream tasks such as ranging and navigation.

Please refer to Figure 2. Figure 2 is a schematic diagram of another application scenario of the 3D model reconstruction method based on multi-source input provided in the embodiments of this application. Specifically:

The 3D model reconstruction platform 12 can be deployed on infrastructure 14 to perform fusion mapping based on multi-source inputs and output mapping results 16. Alternatively, it can be deployed on infrastructure 15 to reconstruct a 3D model 17 based on the mapping results 16 and output the 3D model 17. Furthermore, user A can directly access the 3D model reconstruction platform 12. Specifically, when the 3D model reconstruction platform 12 is mapping the object to be reconstructed, the user can monitor the mapping results in real time and input corresponding commands to control the mapping process. For example, infrastructure 14 is a computing device deployed on the user's end, located close to the object to be reconstructed 10 for local mapping and computation. It also allows the user to view the mapping results 16 in real time. If there are missed areas or missing details, the user can quickly supplement the data to achieve high-precision acquisition. Based on this, infrastructure 15 is deployed in the cloud. The cloud-provided infrastructure 15 provides the computing power for the 3D model reconstruction platform 12 during the 3D model reconstruction process based on the mapping results 16, meeting the computational needs of 3D model reconstruction. Meanwhile, the digital asset 3D model reconstruction platform and the digital twin simulation platform can directly call the 3D model 17 on the cloud, making it more convenient and efficient. In another example, infrastructure 14 and infrastructure 15 can both be computing devices deployed on the user side, that is, the 3D model reconstruction platform 12 runs directly on the user-side device, thereby enabling the user to have full control over the process of fusion mapping and 3D model reconstruction on the 3D model reconstruction platform 12.

For the object 10 to be reconstructed, data acquisition device 11 acquires data from the object 10 to obtain its multi-source input data. The acquisition device 11 can be various mapping devices such as robots, professional data acquisition equipment, and portable handheld devices, or it can include sensors such as LiDAR, IMU, and RGB (RGB Color Mode) cameras required for high-precision mapping. Spatiotemporal alignment of data between all sensors is achieved through methods such as extrinsic parameter calibration of LiDAR and cameras, establishment of LiDAR and IMU, and hardware-triggered timestamp synchronization of sensors. It may also include a computing unit (responsible for data acquisition and deployment and operation of the high-precision mapping module), a storage unit (responsible for storing and managing multimodal datasets), and a network unit (responsible for high-speed data transmission). The acquisition devices mentioned above can be one or more combined and acquired simultaneously; this application does not limit this. Therefore, when the acquisition device 11 acquires the corresponding data, it obtains raw data such as initial image information, initial point cloud information, and inertial information. In order to facilitate data processing during the mapping process of the 3D model reconstruction platform 12, a unified data structure and a unified application programming interface (API) can be used during the acquisition process to improve mapping efficiency.

To facilitate a clearer explanation of the embodiments of this application, the 3D Gaussian model will be used as an example of a three-dimensional model in the following embodiments. The 3D Gaussian model is an example of a three-dimensional model protected by this application. As shown in the definition of a three-dimensional model, there are still many other models that can achieve three-dimensional reconstruction of a scene in the field of three-dimensional model reconstruction. This should not be construed as a specific limitation.

Please refer to Figure 3. As shown in Figure 3, Figure 3 is a flowchart illustrating a three-dimensional model reconstruction method based on multi-source input provided in an embodiment of this application. Specifically:

Step S201. Obtain the mapping results of the object to be reconstructed. The mapping results include the image information and point cloud information of the object to be reconstructed.

The 3D model reconstruction platform 12 acquires the mapping results of the object to be reconstructed, which serve as input for the 3D model reconstruction. The mapping results can be provided by the user using different methods based on their business needs and the characteristics of the object to be reconstructed; this application does not limit this. It is worth noting that the mapping results include image information and point cloud information of the object to be reconstructed. The image information can be the initial image information of the object to be reconstructed acquired by the acquisition device, or it can be specific image information after fusion and fitting, reflecting information such as the color and shape of the object to be reconstructed. The point cloud information, exemplarily, can be laser point cloud information. Laser point cloud information can accurately determine the depth and geometric information of the object to be reconstructed. Compared to shapes determined by structured light or pure vision methods, the embodiments of this application have higher precision and accuracy.

In one embodiment of this application, the mapping result can be generated by the 3D model reconstruction platform 12. For specific steps in step S201, please refer to Figure 4. Figure 4 is another flowchart illustrating the 3D model reconstruction method based on multi-source input provided in this embodiment of the application. Specifically:

Step S301. Obtain the initial image information and initial point cloud information of the object to be reconstructed, as well as the image inertial information and point cloud inertial information corresponding to the acquisition device when acquiring the initial image information and the acquisition device when acquiring the initial point cloud information.

The 3D model reconstruction platform 12 acquires the initial image information and initial point cloud information of the object to be reconstructed through the acquisition device 11, as well as the image inertial information and point cloud inertial information corresponding to the acquisition device when acquiring the initial image information and the acquisition device when acquiring the initial point cloud information.

Specifically, please refer to Figure 5. Figure 5 is a schematic diagram of a mapping process for a 3D model reconstruction method based on multi-source input provided in the application embodiment, wherein:

Step S3011. Obtain the initial point cloud information of the object to be reconstructed.

The 3D model reconstruction platform 12 obtains the initial point cloud information of the object to be reconstructed by the data reported by the acquisition device 11. For example, the initial point cloud information can be a set obtained by the acquisition device 11, such as a lidar, according to a certain acquisition path and corresponding pose. The initial point cloud information includes frame-by-frame point cloud information.

Step S3012. Obtain the inertial information of the object to be reconstructed.

The 3D model reconstruction platform 12 acquires the inertial information of the object to be reconstructed during the acquisition of initial point cloud information and initial image information by collecting data reported by the acquisition device 11. The acquisition device 11 can simultaneously integrate a LiDAR, an RGB camera, and an IMU to achieve synchronous data acquisition and improve acquisition accuracy. The inertial information of the object to be reconstructed is also the prior motion information for acquiring the initial point cloud information and initial image information. This prior motion information can include angular velocity, acceleration, and other information.

Step S3013. Obtain the initial image information of the object to be reconstructed.

The 3D model reconstruction platform 12 obtains the initial image information of the object to be reconstructed by the data reported by the acquisition device 11. This initial image information can be acquired by the acquisition device 11, such as an RGB camera, including frame-by-frame images of the object to be reconstructed during the acquisition process.

Step S302. Generate mapping results based on initial image information, initial point cloud information, point cloud inertial information, and image inertial information.

For example, the 3D model reconstruction platform 12 integrates frame-by-frame laser point clouds with motion prior information provided by the IMU. Furthermore, it integrates image features with the motion prior information provided by the IMU to jointly estimate the pose state of the entire initial image information and initial point cloud information, generating the mapping result.

Step S3024. Generate point cloud pose information.

The 3D model reconstruction platform 12 integrates frame-by-frame laser point cloud with motion prior information provided by IMU, matches the features of adjacent point clouds, and estimates the point cloud pose information in real time.

Step S3025. Determine descriptor information based on initial image information.

The 3D model reconstruction platform 12 determines the descriptive information of each frame of the image based on the initial image information reported by the acquisition device 11. The descriptive information can be point-n-point (PnP) descriptive information.

Specifically, please refer to Figure 6. As shown in Figure 6, Figure 6 is a schematic diagram of a keyframe determination process for a 3D model reconstruction method based on multi-source input provided in an embodiment of this application. The steps are as follows:

Step S3025. Determine descriptor information based on the initial image information.

By defining descriptor information, the 3D model reconstruction platform 12 can perform frame-by-frame judgment on the initial image information and determine the image's descriptor information frame by frame. For example, the descriptor information is a PnP descriptor. Specifically, the PnP descriptor information may include: PnP residual: an image pose reliability index; a high residual indicates unreliable pose accuracy; and the number of PnP pairs: an image texture feature richness index; a low number of pairs indicates severe visual degradation. According to this preset rule, the 3D model reconstruction platform 12 performs frame-by-frame judgment on the initial image information. This preset rule can also be configured by the user to filter images that conform to specific rules.

This step allows for the filtering of input data, especially for selecting relevant image data that meets specific requirements, thus providing specific images for subsequent mapping and 3D model reconstruction.

Step S303. Determine the keyframe image based on the initial image information and descriptor information.

For example, by using preset rules for descriptor information, such as if the PnP residual in the user-input PnP descriptor is less than a first threshold and/or the number of PnP point pairs is greater than a second threshold, the 3D model reconstruction platform 12 filters the initial image information frame by frame according to the descriptor information. The images that meet the descriptor information after filtering have pose accuracy and image texture feature richness that meet the PnP descriptor requirements and are regarded as keyframe images.

In this example, by filtering the initial image information, keyframe images with rich visual features and accurate poses are retained, while images with severe visual degradation and large pose errors are discarded. This allows the 3D model reconstruction platform 12 to eliminate interfering images and improve mapping accuracy during the mapping process. Furthermore, by configuring keyframe images during 3D model reconstruction, optical and geometric residuals can be verified, thereby improving the accuracy and precision of the 3D model reconstruction.

Step S3026. Generate image pose information.

The 3D model reconstruction platform 12 integrates image features and motion prior information provided by the IMU to estimate the spatiotemporal displacement relationship between two adjacent frames and output the image pose.

Step S3027. The 3D model reconstruction platform 12 performs loop closure detection.

Because IMUs accumulate measurement errors during long-term operation, misalignment may occur between the current sensor information and the already constructed point cloud map when the data acquisition device returns to the mapped area after a prolonged period of operation. Therefore, loop closure detection is necessary during the mapping process to check whether the current frame can form a loop with historical information and to construct a pose map. This improves the accuracy of the mapping data and ensures mapping precision.

For example, a robot moves along a corridor, enters a room, and then returns to the corridor. The robot continuously acquires images and records its pose (position and orientation). Each image is processed to extract key features, such as oriented fast and rotated briefs (ORBs). These feature points can be represented using feature descriptors. As the robot moves, newly acquired images are compared with previously acquired images using feature matching. A feature matching algorithm is used to find similar feature points. If a sufficient number of matching feature points are found (e.g., exceeding a certain threshold), a potential loop closure is considered to exist. For each loop closure candidate, the corresponding image and robot pose are recorded. A pose graph is constructed, where nodes represent robot poses and edges represent relative transformations between adjacent poses. For each loop closure candidate, a constraint edge is added, indicating that the robot returns to its previous position. The feature matching results of the loop closure candidates and the robot pose information are used to verify the authenticity of the loop closure. Further validation can be achieved by calculating the geometric consistency among loop closure candidates, for example, by using a Random Sample Consensus (RANSAC) algorithm to estimate the optimal relative pose. Once a loop closure is confirmed, the pose graph needs to be updated to reflect the latest loop closure information. Graph optimization algorithms are used to minimize errors in the pose graph and maintain map consistency.

Step S3028. The 3D model reconstruction platform 12 performs backend optimization.

After loop closure detection is completed, the 3D model reconstruction platform 12 performs closed-loop optimization on the mapping results based on the matching information of the current loop closures, eliminating mapping misalignment caused by sensor errors. The matching results are optimized through constraints from point cloud information and image information.

Step S3029. The 3D model reconstruction platform 12 generates the mapping results.

For example, after the above steps are completed, the 3D model reconstruction platform 12 can match the corresponding initial point cloud information and initial image information with the image pose information and point cloud pose information to generate the corresponding mapping result. The mapping result includes image information and point cloud information. The image information can be a set of images from the initial image information, and the point cloud information can be a set of point clouds from the initial point cloud information. The image information includes keyframe images. As an example of this application, the 3D model reconstruction platform 12 does not determine descriptor information and does not identify keyframe images; the image information in the mapping result is only a subset of the initial image information.

It is worth noting that the above-described mapping process steps S3011-S3029 are one embodiment of this application, and any combination of steps or any execution order can fall within the protection scope of this application. In other embodiments, such as steps S3027, S3025, and S3028, steps can be additional steps for the 3D model reconstruction platform 12 to build a map based on the initial image information, initial point cloud information, and corresponding initial inertial information.

By adopting a multi-source input approach for mapping the object to be reconstructed, mapping efficiency can be greatly improved. Point cloud information can provide more accurate geometric and depth information of the object to be reconstructed. At the same time, the introduction of IMU inertial information can more quickly and accurately realize the input of motion prior information when the acquisition device acquires initial point cloud information and initial image information. In the subsequent mapping process, the accuracy of pose matching and fitting is higher, avoiding the problem of identifying relatively coarse geometric and inertial information through a lot of calculations based solely on image information, thus greatly improving mapping efficiency and accuracy.

Based on this, in one embodiment of this application, the 3D model reconstruction platform 12 can provide real-time mapping feedback to the user during the mapping process. When data omissions, image loss, or loss of details occur, the user can promptly take remedial measures such as supplementary data acquisition. Specifically:

Step S3023. The 3D model reconstruction platform 12 obtains the supplementary sampling command.

For example, the 3D model reconstruction platform 12 waits for the acquisition device 11 to report the supplementary acquisition data based on the supplementary acquisition command input by user A. This data includes supplementary image information, supplementary point cloud information, supplementary image inertial information, and supplementary point cloud inertial information. Based on this supplementary acquisition data, the platform updates the mapping results. This improves mapping efficiency and ensures the integrity of the mapping results for the object to be reconstructed.

For example, the 3D model reconstruction platform 12 provides some interfaces that allow users to customize queries for relevant content.

GetCameraInfo()CameraInfo: Gets the camera parameters and status at the current moment.

Object definition: current image frame timestamp, image number, image intrinsic parameters, image pose information, whether it is a keyframe, and image data.

GetLidarInfo()LidarInfo: Gets the current LiDAR parameters and status.

Object definition: LiDAR type, including solid-state LiDAR and multi-line LiDAR; number of LiDAR lines, LiDAR current frame timestamp, LiDAR point cloud frame number, LiDAR pose information, whether a loop closure is detected at the current position, the LiDAR point cloud frame number associated with the loop closure, and the original LiDAR point cloud data.

GetIMUInfo(): Gets the IMU parameters and status at the current time.

Object definition: IMU type, including six-axis and nine-axis; IMU timestamp and IMU data.

GetMapInfo()MapInfo: Retrieves the global point cloud mapping result at the current moment.

Object definition: point cloud coordinates, point cloud color value.

Interfaces that rely on user input include:

Set the PnP descriptor threshold for automated keyframe extraction and input parameters:

Residual: PnP residual threshold;

PairNum: Threshold for the number of PnP point pairs.

It is worth noting that the above mapping process is one embodiment of this application. It should be understood that the above mapping steps do not limit the mapping results in this application during the 3D model reconstruction process.

Step S202. Establish a three-dimensional model of the object to be reconstructed based on the mapping results.

In one embodiment of this application, taking a 3D Gaussian model as an example, the 3D model reconstruction platform 12 establishes a 3D Gaussian model based on the mapping results. This 3D Gaussian model typically includes the following five parameters:

Location: also known as mean, represents the coordinates of the center of the 3D Gaussian kernel in three-dimensional space;

Covariance: Represents the shape distribution of the 3D Gaussian kernel. The three column vectors in the covariance matrix represent the three principal axis directions of the Gaussian ellipsoid.

Scaling factor: Represents the size of each 3D Gaussian kernel;

Opacity: Represents the transparency information of the 3D Gaussian kernel. The higher the opacity, the closer the Gaussian kernel is to the object's surface.

Lighting parameters: Encode lighting information from different viewpoints, which can reflect the changes in lighting within a 3D scene.

For example, feature points or landmarks are selected from the mapping results. These feature points can be fixed points in the environment, such as markings on walls, corners of rooms, etc. Based on this, the mean vector μ of each feature point is determined, where μ is typically the estimated position of the feature point in the world coordinate system. Further, the covariance matrix Σ is determined, which describes the uncertainty of the feature point's position. A 3D Gaussian model is then built. For each selected feature point, a 3D Gaussian distribution model is constructed using the mean vector and covariance matrix determined above to complete the 3D Gaussian model construction.

Step S203. Determine the target image information and target point cloud information.

Specifically, the target image information and target point cloud information of the object to be reconstructed are determined from the mapping results under the observation of the target pose. The target pose includes position information and attitude information. The target pose can be the position and attitude information used to observe the target image information and target point cloud information in the mapping results. This target pose can be input by user A, or it can be selected by the 3D model reconstruction platform 12 according to preset rules, such as setting random rules, and the 3D model reconstruction platform 12 determines the target pose according to the preset rules.

Step S2031. Determine the target pose based on the keyframe image.

In one embodiment of the present invention, the target pose can be determined based on keyframe images determined by the 3D model reconstruction platform 12 during the mapping process. Specifically, when the keyframe images are determined based on descriptor information, they carry corresponding pose information. When determining the target pose, the pose information of the keyframe images can be directly determined as the target pose. As mentioned above, keyframe images are usually determined through descriptor information and have the characteristics of high image texture feature richness and accurate pose. Therefore, by determining the target pose based on keyframe images, the efficiency and effectiveness of subsequent 3D model adjustment and training can be improved.

Step S204. Instruct the 3D model to generate a target depth image and a target digital image based on the target pose.

The 3D model reconstruction platform 12 instructs the 3D Gaussian model to generate a target depth image and a target digital image based on the target pose. For example, taking the target digital image as an example, based on the target pose, the coordinates of the Gaussian model are transformed from the world coordinate system to the corresponding coordinate system. The position of the 3D Gaussian model in the target pose coordinate system is projected onto the 2D image plane. Then, the excess portion is cropped according to the image boundary. Points are sampled on the image plane, and the probability of these points corresponding to the 3D Gaussian model is calculated. The sampled probability values are converted into grayscale or color values to obtain the final image.

Step S205. Determine the geometric residual based on the target point cloud information and the target depth image.

The 3D model reconstruction platform 12 determines the geometric residual based on the target point cloud information of the object to be reconstructed under the target pose observation in the mapping results, and the target depth image generated by the 3D Gaussian model based on the target pose. For example, the 3D model reconstruction platform 12 can determine the target point cloud information corresponding to the target pose based on the target pose, compare the target point cloud information with the target depth image, and then calculate the distance between these point clouds and the expected position of the Gaussian distribution to determine the geometric residual.

Step S206. Determine the optical residual based on the target image information and the target digital image.

The 3D model reconstruction platform 12 determines the optical residual based on the target image information of the object to be reconstructed under the target pose observation in the mapping results, and the target image information generated by the 3D Gaussian model based on the target pose. For example, the 3D model reconstruction platform 12 can determine the target image information corresponding to the target pose in the target image information based on the target pose. If the target pose is the pose information of a keyframe image, then the target image information is the keyframe image. The residual can be defined as the difference between the actual pixel value and the rendered pixel value. The optical residual is determined by comparing the target image information and the target digital image.

Step S207. Adjust the 3D model based on the geometric residuals and optical residuals.

Furthermore, the 3D model reconstruction platform 12 adjusts the 3D Gaussian model based on the optical and geometric residuals obtained in the above steps. For example, a nonlinear optimization algorithm, such as gradient descent, is used to minimize the geometric and/or optical residuals while updating the parameters of each Gaussian distribution until convergence or the maximum number of iterations is reached.

Therefore, by utilizing the image information and point cloud information included in the mapping results generated from multiple inputs, and constructing optical and geometric residuals for the target depth image and target digital image generated based on the target pose, the optical and geometric residuals are simultaneously introduced to adjust the 3D model. In particular, the geometric residuals are introduced through point cloud information to supervise the training of the 3D model, so that the 3D model, after adjustment, is based on real scale information and has higher geometric accuracy and richer detail features.

Step S208. Perform anti-aliasing calculations based on the target pose.

For example, during the rendering of a 3D Gaussian model, rendering anomalies occur when there is a significant discrepancy between the resolution of the user-input rendering viewpoint and the training viewpoint. This is because the opacity parameter trained at the original resolution is only valid for that resolution. When the user performs a significant scaling operation, the opacity parameter error will cause abnormal occlusion within the image plane. Based on this, one embodiment of this application specifically calculates an opacity compensation coefficient. Where Σ is the covariance matrix, I is the identity matrix, and the opacity compensation coefficient ρ is calculated at the current pose to compensate for the opacity of the target digital image during the 3D Gaussian model rendering process. By compensating for the opacity parameter according to the current resolution, the rasterization framework can correctly render texture details at different resolutions.

Step S209. The 3D model reconstruction platform 12 instructs the 3D model to adjust its pose during the rendering process.

In one embodiment of this application, by configuring the rasterization rendering function to be differentiable relative to the image pose during rasterization rendering, and using the gradient information of geometric and optical residuals, image pose supervision is achieved. When the image pose shifts beyond a certain threshold, the image pose is adjusted to reduce the error. As a result, ghosting phenomena appearing in the 3D Gaussian model reconstruction results are reduced, and rendering quality is further improved.

The 3D model reconstruction platform 12 can provide users with corresponding interfaces for configuring and controlling the 3D model reconstruction process, for example:

GetGaussMapInfo()GaussMapInfo: Gets the parameters of a 3D Gaussian model.

Object definition: Gaussian kernel center position, covariance, scaling factor, opacity, spherical harmonic function parameters.

GetTrainingInfo(): Gets the current status and feedback of the 3D Gaussian model training task.

Object definitions: iteration steps, optical residual, geometric residual, peak signal-to-noise ratio (PSNR) (used to evaluate image rendering quality), and structural similarity (used to evaluate the similarity between the rendered image and the ground truth).

GetCameraOptimizationInfo()CameraOptimizationInfo: Gets the image pose optimization result.

Object definition: Keyframe number, keyframe timestamp, original pose, optimized image pose, image pose correction magnitude

Interfaces that rely on user input include:

SetConfig(Params, UseGeoLoss, UseCameraOpt, UseAntiAliasing): Sets the training parameters and input parameters for the 3D Gaussian reconstruction module.

Params: Includes basic parameter configurations such as training iteration steps and optimizer learning rate. The reconstruction framework provides recommended settings based on the scale of the reconstructed scenario for user reference.

UseGeoLoss: Whether to use geometric residual supervision;

UseCameraOpt: Whether to enable camera extrinsic parameter joint optimization;

UseAntiAliasing: Whether to enable anti-aliasing calculation.

Based on this, in one embodiment of this application, referring to Figure 1, user A can configure whether to enable anti-aliasing calculation, the number of training iterations, etc., through the 3D model reconstruction platform 12. For example, user A inputs the number of iterations, and the 3D model reconstruction platform 12 instructs the number of adjustments to the 3D model according to the number of iterations input by user A. After the 3D model has undergone the number of iterations based on the geometric residuals and optical residuals, it outputs the adjusted 3D model.

This application embodiment also provides a three-dimensional model reconstruction platform 12. Please refer to Figure 8 below. Figure 8 is a structural schematic diagram of a three-dimensional model reconstruction platform provided in this application embodiment. Specifically, it includes the following modules:

The acquisition module 901 is used to acquire the mapping result of the object to be reconstructed, wherein the mapping result includes the image information and point cloud information of the object to be reconstructed;

Module 903 is used to create a 3D model of the object to be reconstructed based on the mapping results.

The determination module 902 is also used to determine the target image information and target point cloud information of the object to be reconstructed under the target pose observation from the mapping results. The target pose includes position information and attitude information.

Instruction module 904 is used to instruct the 3D model to generate a target depth image and a target digital image based on the target pose;

The determination module 902 is used to determine the geometric residual based on the target point cloud information and the target depth image; the determination module 902 is also used to determine the optical residual based on the target image information and the target digital image.

Adjustment module 905 is used to adjust the three-dimensional model based on geometric residuals and optical residuals.

In another embodiment of this application, the three-dimensional model reconstruction platform 12 may further include:

The generation module 906 is used to generate mapping results based on initial image information, initial point cloud information, point cloud inertial information, and image inertial information.

The update module 907 is used to update the mapping results based on the supplemented image information, supplemented point cloud information, supplemented laser inertial information, and supplemented image inertial information.

Output module 908 is used to output a 3D model after the number of iterations is satisfied.

It is worth noting that the user in the above embodiments can be replaced by multiple other users, and the above modules can all implement the corresponding technical functions. This application embodiment does not limit this.

This application embodiment provides an example of obtaining module 901, determining module 902, establishing module 903, instructing module 904, and adjusting module 905. Similarly, the implementation of generating module 906, updating module 907, and output module 908 can refer to the implementation of the aforementioned modules.

Specifically, the acquisition module 901, the determination module 902, the establishment module 903, the indication module 904, and the adjustment module 905 can all be implemented in software or in hardware. For example, the implementation of the acquisition module 901 will be described below. Similarly, the implementation of the determination module 902, the establishment module 903, the indication module 904, and the adjustment module 905 can refer to the implementation of the acquisition module 901.

As an example of a software functional unit, module 901 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, module 901 may include code running on multiple hosts/virtual machines/containers. It should be noted that the multiple hosts/virtual machines/containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts/virtual machines/containers used to run the code may be distributed in the same availability zone or in different availability zones, each availability zone including one data center or multiple geographically proximate data centers. Typically, a region may include multiple availability zones.

As an example of a hardware functional unit, the acquisition module 901 may include at least one computing device, such as a server. Alternatively, the acquisition module 901 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

It should be noted that, in other embodiments, the acquisition module 901 can be used to execute any step in the multi-source input-based 3D model reconstruction method, the determination module 902 can be used to execute any step in the multi-source input-based 3D model reconstruction method, the establishment module 903 can be used to execute any step in the multi-source input-based 3D model reconstruction method, the instruction module 904 can be used to execute any step in the multi-source input-based 3D model reconstruction method, and the adjustment module 905 can be used to execute any step in the multi-source input-based 3D model reconstruction method. The steps implemented by the acquisition module 901, determination module 902, establishment module 903, instruction module 904, and adjustment module 905 can be specified as needed. By implementing different steps in the multi-source input-based 3D model reconstruction method through the acquisition module 901, determination module 902, establishment module 903, instruction module 904, and adjustment module 905, all functions of the 3D model reconstruction platform can be realized.

The methods, three-dimensional model reconstruction platforms, and systems of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above-described solutions of the embodiments of this application, relevant equipment for cooperating in the implementation of the above solutions is also provided below.

This application provides a computing device. Please refer to Figure 9 below. Figure 9 is a schematic diagram of the structure of a computing device according to an embodiment of this application for a three-dimensional model reconstruction method based on multi-source input. The computing device 900 includes: a bus 911, a processor 912, a memory 910, and a communication interface 909. The processor 912, the memory 910, and the communication interface 909 communicate via the bus 911. The computing device 900 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

Bus 911 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not indicate that there is only one bus or one type of bus. Bus 911 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 910, processor 912, communication interface 909).

The processor 912 may include any one or more of the following processors: a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

The memory 910 may include volatile memory, such as random access memory (RAM). The processor 912 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

The memory 910 stores executable program code, which the processor 912 executes to implement the functions of the acquisition module 901, the determination module 902, the establishment module 903, the instruction module 904, and the adjustment module 905, thereby realizing the 3D model reconstruction method based on multi-source input. In other words, the memory 910 stores instructions for the 3D model reconstruction platform to execute the 3D model reconstruction method based on multi-source input.

The communication interface 909 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

Please refer to Figure 10 below. Figure 10 is a schematic diagram of a computing device cluster for a multi-source input-based 3D model reconstruction method according to an embodiment of this application. As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 910 of one or more computing devices 900 in the computing device cluster may store the same 3D model reconstruction platform for executing instructions of the multi-source input-based 3D model reconstruction method.

In some possible implementations, one or more computing devices 900 in the computing device cluster can also be used to execute some of the instructions of the 3D model reconstruction platform for executing a 3D model reconstruction method based on multi-source input. In other words, a combination of one or more computing devices 900 can jointly execute the instructions of the 3D model reconstruction platform for executing a 3D model reconstruction method based on multi-source input.

It should be noted that the memory 910 in different computing devices 900 within the computing device cluster can store different instructions for executing some functions of the 3D model reconstruction platform. That is, the instructions stored in the memory 910 of different computing devices 900 can implement the functions of one or more modules among the acquisition module 901, determination module 902, creation module 903, instruction module 904, and adjustment module 905.

In some possible implementations, the memory 910 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the multi-source input-based 3D model reconstruction method. In other words, a combination of one or more computing devices 900 can jointly execute instructions for executing the multi-source input-based 3D model reconstruction method.

Please refer to Figure 11 below. Figure 11 is another schematic diagram of the computing device cluster of the multi-source input-based 3D model reconstruction method according to an embodiment of this application. As shown in Figure 11, two computing devices 900A and 900B are connected through a communication interface 909. The memory in computing device 900A stores instructions for executing the determination module 902, the establishment module 903, and the adjustment module 905. The memory in computing device 900B stores instructions for the acquisition module 901 and the instruction module 904 for the functions to be executed. In other words, the memory 910 of computing devices 900A and 900B jointly stores the instructions used by the 3D model reconstruction platform to execute the multi-source input-based 3D model reconstruction method.

The connection method between the computing device clusters shown in Figure 11 can be considered because the 3D model reconstruction method based on multi-source input provided in this application requires a large amount of data transmission to the acquisition module 901. Considering the amount of data transmission, in order to avoid overloading the computing device 900A, the functions to be implemented are delegated to the computing device 900B.

It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

Please refer to Figure 12 below. Figure 12 is another schematic diagram of the computing device cluster structure of the 3D model reconstruction method based on multi-source input according to an embodiment of this application. In some possible implementations, one or more computing devices in the computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 shows one possible implementation, where two computing devices 900C and 900D are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this type of possible implementation, the memory 910 in computing device 900C stores instructions for executing the determination module 902, the establishment module 903, and the adjustment module 905. Simultaneously, the memory 910 in computing device 900D stores instructions for executing the acquisition module 901 and the instruction module 904.

The connection method between the computing device clusters shown in Figure 12 can be considered as follows: the three-dimensional model reconstruction method based on multi-source input provided in this application requires a large amount of data transmission and needs to be connected through a network. The execution of these functions is relatively independent. In order to achieve the best storage and computing performance, the data transmission function is considered to be executed by the computing device 900D.

It should be understood that the functions of computing device 900C shown in Figure 12 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900D can also be performed by multiple computing devices 900.

In some possible implementations, the memory 910 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the multi-source input-based 3D model reconstruction method. In other words, a combination of one or more computing devices 900 can jointly execute instructions for executing the multi-source input-based 3D model reconstruction method.

This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to execute the above-described method for performing multi-source input-based 3D model reconstruction on a 3D model reconstruction platform.

This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the above-described method applied to a 3D model reconstruction platform for performing a 3D model reconstruction based on multi-source input.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Those skilled in the art will clearly understand that the specific working process of the system, 3D model reconstruction platform or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to execute the above-described method for performing multi-source input-based 3D model reconstruction on a 3D model reconstruction platform.

This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute the above-described method applied to a 3D model reconstruction platform for performing a 3D model reconstruction based on multi-source input.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Those skilled in the art will clearly understand that the specific working process of the system, 3D model reconstruction platform or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims (15)

A method for reconstructing a 3D model based on multi-source input, characterized in that the method is applied to a 3D model reconstruction platform, and the method includes: Obtain the mapping result of the object to be reconstructed, wherein the mapping result includes the image information of the object to be reconstructed and the point cloud information of the object to be reconstructed; A three-dimensional model of the object to be reconstructed is established based on the mapping results. The target image information and target point cloud information of the object to be reconstructed under the target pose observation are determined from the mapping results, wherein the target pose includes position information and attitude information; The three-dimensional model is instructed to generate a target depth image and a target digital image based on the target pose; The geometric residual is determined based on the target point cloud information and the target depth image; The optical residual is determined based on the target image information and the target digital image; The three-dimensional model is adjusted based on the geometric residuals and the optical residuals. The method according to claim 1, wherein obtaining the mapping result of the object to be reconstructed includes: Acquire the initial image information and initial point cloud information of the object to be reconstructed, as well as the image inertial information and point cloud inertial information corresponding to the acquisition device when acquiring the initial image information and the acquisition device when acquiring the initial point cloud information; The mapping result is generated based on the initial image information, the initial point cloud information, the point cloud inertial information, and the image inertial information. The method according to claim 1 or 2, characterized in that the method further comprises: Descriptor information is determined based on the initial image information, wherein the descriptor information includes texture features of at least one frame of the initial image information; Based on the initial image information and the descriptor information, the keyframe image is determined. The method according to claim 3, characterized in that the method further comprises: Based on the keyframe image, determine the target pose for observing the keyframe image. The method according to any one of claims 1 to 4, characterized in that the method further comprises: Acquire the supplementary image information, supplementary point cloud information, supplementary image inertial information, and supplementary laser inertial information of the object to be reconstructed; The mapping result is updated based on the supplemented image information, the supplemented point cloud information, the supplemented laser inertial information, and the supplemented image inertial information. The method according to any one of claims 1 to 5, characterized in that the method further comprises: Obtain the iteration count input by the user, which indicates the number of adjustments made to the 3D model; Once the required number of iterations is met, the 3D model is output. A three-dimensional model reconstruction platform, characterized in that the method of the three-dimensional model reconstruction platform includes: The acquisition module is used to acquire the mapping result of the object to be reconstructed, the mapping result including the image information of the object to be reconstructed and the point cloud information of the object to be reconstructed; A modeling module is provided, which is used to create a three-dimensional model of the object to be reconstructed based on the mapping results. The determination module is used to determine the target image information and target point cloud information of the object to be reconstructed under the target pose observation from the mapping result, wherein the target pose includes position information and attitude information; The instruction module is used to instruct the 3D model to generate a target depth image and a target digital image based on the target pose; The determining module is further configured to determine the geometric residual based on the target point cloud information and the target depth image; The determining module is further configured to determine the optical residual based on the target image information and the target digital image; An adjustment module is used to adjust the three-dimensional model based on the geometric residual and the optical residual. According to claim 7, the three-dimensional model reconstruction platform is characterized in that the acquisition module is used to acquire the mapping result of the object to be reconstructed, then: The acquisition module is specifically used to acquire the initial image information and initial point cloud information of the object to be reconstructed, as well as the image inertial information corresponding to when the acquisition device acquires the initial image information and the point cloud inertial information corresponding to when the initial point cloud information is acquired; the 3D model reconstruction platform further includes: A generation module is used to generate the mapping result based on the initial image information, the initial point cloud information, the point cloud inertial information, and the image inertial information. The three-dimensional model reconstruction platform according to claim 7 or 8 is characterized in that, The determining module is further configured to determine descriptor information based on the initial image information, wherein the descriptor information includes the texture features of at least one frame of the initial image information; The determining module is further configured to determine the keyframe image based on the initial image information and the descriptor information. The three-dimensional model reconstruction platform according to claim 9 is characterized in that, The acquisition module is further configured to determine the target pose for observing the keyframe image based on the keyframe image. The three-dimensional model reconstruction platform according to any one of claims 7 to 10 is characterized in that, The acquisition module is further configured to acquire supplementary image information, supplementary point cloud information, supplementary image inertial information, and supplementary laser inertial information of the object to be reconstructed; the three-dimensional model reconstruction platform further includes: An update module is used to update the mapping result based on the supplemented image information, the supplemented point cloud information, the supplemented laser inertial information, and the supplemented image inertial information. The three-dimensional model reconstruction platform according to any one of claims 7 to 11 is characterized in that: The acquisition module is further configured to acquire the number of iterations input by the user, the number of iterations indicating the number of adjustments to the 3D model; the 3D model reconstruction platform further includes: An output module is used to output the three-dimensional model after the number of iterations is satisfied. A computing device cluster, characterized in that it includes at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 6. A computer program product containing instructions, characterized in that, when the instructions are executed by a cluster of computer devices, the cluster of computer devices causes the cluster of computer devices to perform the method as described in any one of claims 1 to 6. A computer-readable storage medium, characterized in that it includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 6.