TWI846598B - 3d surface reconstruction method - Google Patents

Abstract

The application provides a 3D surface reconstruction method, including: extracting multiple visual features from multiple multi-view images of an object and obtaining multiple camera pose information; converting the camera pose information into multiple posed embedding information; adding the visual features with the posed embedding information to generate input information; inputting the input information into an encoder of a transformer model, and sequentially performing a first attention operation from sequence to sequence to output volume features; inputting the volume features into a decoder of the transformer model, several layers of second attention operations are performed sequentially to generate a feature prediction result; mapping an operation dimension of the feature prediction result into a one-dimensional feature dimension; and reconstructing a three-dimensional model of the object according to the one-dimensional feature dimension.

Description

Three-dimensional surface reconstruction method

This case is about a three-dimensional surface reconstruction method using a neural network architecture of the Transformer model.

Existing 3D surface reconstruction methods usually first capture multi-view images through a camera and combine them with a camera simultaneous localization and mapping (SLAM) system to obtain camera pose information. Then, the images from different viewpoints are used to generate depth maps using the depth map estimation method, and the feature pixels between the images are matched using the stereo matching algorithm. Finally, the depth map, matched feature pixels and camera pose information are combined into a three-dimensional model representation format such as a point cloud, polygon network or truncated signed distance function (TSDF).

However, the aforementioned methods usually face two major difficulties. The first is that the depth maps to be fused into the three-dimensional model are from different perspectives and have different camera postures, and they cannot be smoothly stitched and fused. The second is that when the texture of the image is too smooth or repetitive, it will make it difficult for the stereo matching algorithm to find the correct matching feature pixels. Therefore, the two difficulties faced by the aforementioned reconstruction methods will lead to many reconstruction defects on the surface of the reconstructed three-dimensional model.

This case provides a three-dimensional surface reconstruction method, including: extracting multiple visual features from multiple multi-view images of an object and obtaining multiple camera posture information. Converting the camera posture information into multiple posture embedding information. Adding the visual features and the posture embedding information to generate multiple input information. Sending the input information to an encoder of a converter model, and sequentially performing several layers of sequence-to-sequence first attention operations to output multiple volume features. Inputting the volume features into a decoder of the converter model, and sequentially performing several layers of second attention operations to generate a feature prediction result. Mapping the operation dimension of the feature prediction result into a one-dimensional feature dimension. Finally, reconstructing a three-dimensional model of the object based on the feature dimension.

In summary, this case is a 3D surface reconstruction method that uses the Transformer model network architecture to learn the characteristics of global long range dependency, replacing the method of learning to estimate depth maps at different view angles, and at the same time refers to images at different view angles to update the volumetric representation, thereby improving the reconstruction results of the surface details of the object. Therefore, this case can reconstruct the surface of the object at different view angles and effectively improve the surface reconstruction defects.

The following will be used in conjunction with the relevant drawings to illustrate the embodiments of the present invention. In addition, the drawings in the embodiments omit some components or structures to clearly show the technical features of the present invention. In these drawings, the same reference numerals represent the same or similar components or circuits. It must be understood that although the terms "first", "second", etc. may be used in this article to describe various components, parts, regions or functions, these components, parts, regions and/or functions should not be limited by these terms. These terms are only used to separate one component, component, region or function from another component, component, region or function.

As shown in FIG. 1 , the three-dimensional surface reconstruction method mainly includes steps S10 to S22. First, as shown in step S10, a convolutional neural network (CNN) feature extractor is used to extract multiple visual features from multiple multi-view images of an object, and multiple camera pose information is obtained. The camera pose information includes a camera position coordinate and a camera shooting angle. In one embodiment, the feature extractor can use a feature extractor based on deep learning such as VGGNet, ResNet, EfficientNet, MobileNet, Vision Transformer, Swin Transformer, etc.

As shown in step S12, the acquired camera pose information is transformed through a multi-layer perceptron (MLP) to map the camera pose information to dimension N and transform it into complex pose embedding information, so that the dimension N has the same dimension as the visual features of the multi-view image.

As shown in step S14, these visual features and posture embedding information are added to perform feature fusion to generate multiple input information, which can be used as input of a converter model.

Please refer to FIG. 1 and FIG. 2 at the same time. As shown in step S16, the input information is input into an encoder 12 of the transformer model 10, and the first attention operation of the sequence to the sequence is performed in sequence to output a plurality of volume features. In one embodiment, the encoder 12 is composed of a plurality of encoder blocks 14, each of which further includes a first multi-head self-attention (MSA) module 16 and a first feed forward network (FFN) module 18, wherein the first multi-head self-attention module 16 uses the attention mechanism to perform operations, and the first feed forward network module 18 is responsible for continuously strengthening the detailed expression ability of the feature. Based on this, each input information can be converted into three feature vectors (Q vector, K vector and V vector) through the first multi-head self-attention module 16, and the first attention operation is executed, and then the details are enhanced through the first feedforward network module 18 to output a plurality of volume features, which include information such as a depth, a texture and a spatial information of the object. Taking N coding blocks 14 as an example, the first multi-head self-attention module 16 in each coding block 14 will generate three feature vectors from the output of the previous layer and execute the first attention operation. The attention mechanism can refine the image features of the object and improve the feature quality, so that the three-dimensional features of the object at various angles will be extracted under the layer-by-layer attention mechanism and the detail expression ability will be enhanced through the first feedforward network module 18.

Please refer to FIG. 1 and FIG. 2 at the same time. As shown in step S18, the volume feature is input into a decoder 20 of the converter model 10, and several layers of second attention operations are performed in sequence to generate a feature prediction result. In one embodiment, the decoder 20 is composed of a plurality of decoder blocks 22, each of which further includes a second multi-head self-attention (MSA) module 24, an encoder-decoder attention (EDA) module 26, and a second feed-forward network (FFN) module 28, wherein the second multi-head self-attention module 24 and the encoder-decoder attention module 26 both use the attention mechanism for calculation, and the second feed-forward network module 28 is responsible for continuously enhancing the detail expression capability of the features. Based on this, the feature function output by the upper layer is converted into three feature vectors, namely Q (Query) vector, K (Key) vector and V (Value) vector, through the second multi-head self-attention module 24, and the second attention operation is performed, so that the output of the second multi-head self-attention module 24 is used as a first feature vector (V vector), and the encoding and decoding attention module 26 generates a second feature vector (Q vector) and a third feature vector (K vector) according to the volume feature output by the encoding block 14 as input, and performs the second attention operation based on the first feature vector (V vector), the second feature vector (Q vector) and the third feature vector (K vector), and then strengthens the details through the second feedforward network module 28 to output the feature prediction result. Taking N decoding blocks 22 as an example, the second multi-head self-attention module 24 in each decoding block 22 will generate three feature vectors from the output of the previous layer and perform the first second attention operation. The difference is that in the encoding and decoding attention module 26, the volume features of the output of the encoding block 14 will be used as input to generate the second feature vector (Q vector) and the third feature vector (K vector), and the output of the second multi-head self-attention module 24 will be used as the first feature vector (V vector) to perform the second attention operation, and finally the detail expression ability will be enhanced through the second feedforward network module 28.

Among them, the role of the second multi-head self-attention module 24 in the decoding block 22 is also to refine the quality of the feature function (TSDF Queries), and the attention operation of the encoder-decoder attention module 26 across the encoder and decoder uses the expression ability of the volume features of the object to guide the generation of feature prediction results (such as TSDF features) layer by layer, and at the same time maps the operation dimension from the high-level feature dimension to the spatial dimension where the object reconstruction model used in this case is located.

In one embodiment, the first attention operation used in the encoder 12 in the converter model 10 and the second attention operation used in the decoder 20 are both calculated using the following formula (1), wherein Q represents the Q vector (second eigenvector), K represents the K vector (third eigenvector), V represents the V vector (first eigenvector), the superscript T represents the transposed matrix of the vector, and d _k is the dimension of the K vector (Dimension of K). (1)

As shown in step S20, the calculation dimension of the feature prediction result is mapped into a one-dimensional feature dimension of the original dimension using a truncated signed distance function mapper (TSDF Projector). Finally, as shown in step S22, a three-dimensional model of the object is reconstructed according to the feature dimension. The three-dimensional model is a three-dimensional model reconstruction result expressed in a truncated signed distance function (TSDF) format.

Please refer to FIG. 1 and FIG. 3 . The present invention implements the entire 3D surface reconstruction method through a 3D surface reconstruction system 30. The 3D surface reconstruction system 30 includes an image capture device 32, a central processing unit (CPU) 34, and a graphics processing unit (GPU) 36. The image capture device 32 is electrically connected to the CPU 34, and the CPU 34 is electrically connected to the GPU 36. The image capture device 32 captures each view angle of the object to obtain multiple multi-view angle images, and transmits the multi-view angle images to the CPU 34. The CPU 34 has a built-in machine learning (Machine Learning) The machine learning algorithm can be used to execute steps S10 to S22 shown in FIG. 1. Based on this, the central processing unit 34 can perform the three-dimensional surface reconstruction method shown in steps S10 to S22 after receiving the multi-view images, and the image processing part of the process can be transmitted by the central processing unit 34 to the graphics processing unit 36 for processing to speed up the image processing time. After the processing is completed, the graphics processing unit 36 returns the image to the central processing unit 34, and finally the central processing unit 34 outputs the reconstructed three-dimensional model.

In the above-mentioned embodiment, the CPU 34 and the GPU 36 are built into an electronic device 38, and the electronic device 38 can be a personal computer, a laptop or a tablet computer, etc., but the present invention is not limited thereto. In one embodiment, the electronic device 38 uses the CPU 34 to perform calculations. In addition, in other embodiments, an embedded controller (EC), a microprocessor (Microprocessor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a system on a chip (SOC) or other similar components or combinations can also be selected, but the present invention is not limited thereto.

In one embodiment, the image capture device 32 may be a monochrome camera or a color camera, a stereo camera, a digital camera, a digital camera, a depth camera, or other electronic equipment capable of capturing images.

In summary, this case is a 3D surface reconstruction method that uses the Transformer model network architecture to learn the characteristics of global long range dependency, replacing the learned method of estimating depth maps at different view angles, and at the same time refers to images at different view angles to update the volumetric representation to achieve improved reconstruction results of object surface details. Therefore, this case can reconstruct the surface of objects at different view angles and effectively improve the problem of learned surface reconstruction defects.

The embodiments described above are only for illustrating the technical ideas and features of this case. Their purpose is to enable those familiar with this technology to understand the content of this case and implement it accordingly. They cannot be used to limit the patent scope of this case. In other words, any equivalent changes or modifications made according to the spirit disclosed in this case should still be covered by the scope of the patent application of this case.

10: Converter Model

12: Encoder

14: Coding block

16: The first multi-head self-attention module

18: First Feedback Network Module

20:Decoder

22: Decoding block

24: Second multi-head self-attention module

26: Encoding and decoding attention module

28: Second Feedback Network Module

30: 3D surface reconstruction system

32: Image capture device

34:Central Processing Unit

36: Graphics Processor

38: Electronic devices

S10~S22: Steps

FIG. 1 is a schematic diagram of the process of a three-dimensional surface reconstruction method according to an embodiment of the present invention. FIG. 2 is a schematic diagram of the structure of a converter model used in a three-dimensional surface reconstruction method according to an embodiment of the present invention. FIG. 3 is a block diagram of a three-dimensional surface reconstruction system according to an embodiment of the present invention.

S10~S22: Steps

Claims (8)

A three-dimensional surface reconstruction method comprises: extracting a plurality of visual features from a plurality of multi-view images of an object using a convolutional neural network feature extractor and obtaining a plurality of camera posture information; converting the camera posture information into a plurality of posture embedding information through a multi-layer sensor; performing addition operations on the visual features and the posture embedding information through a central processing unit to generate a plurality of input information; inputting the input information Into an encoder of a converter model, several layers of sequence-to-sequence first attention operations are sequentially performed to output multiple volume features; the volume features are input into a decoder of the converter model, several layers of second attention operations are sequentially performed to generate a feature prediction result; the operation dimension of the feature prediction result is mapped into a one-dimensional feature dimension; and a three-dimensional model of the object is reconstructed according to the feature dimension. The three-dimensional surface reconstruction method as described in claim 1, wherein the multi-view images are obtained by photographing the object at each view angle using an image capture device. A three-dimensional surface reconstruction method as described in claim 1, wherein the camera posture information includes a camera position coordinate and a camera shooting angle. A three-dimensional surface reconstruction method as described in claim 1, wherein the encoder comprises a plurality of encoding blocks, each of which further comprises a first multi-head self-attention module and a first feedforward network module, so as to convert each of the input information into three feature vectors and perform the first attention operation through the first multi-head self-attention module, and then enhance the details through the first feedforward network module to output the volume features. A three-dimensional surface reconstruction method as described in claim 1, wherein the decoder comprises a plurality of decoding blocks, each of which further comprises a second multi-head self-attention module, a coding-decoding attention module and a second feedforward network module, so that the feature function output by the upper layer is converted into three feature vectors through the second multi-head self-attention module and the second attention operation is performed to output a first feature vector, and the coding-decoding attention module generates a second feature vector and a third feature vector according to the volume features, and the second attention operation is performed according to the first feature vector, the second feature vector and the third feature vector, and the details are enhanced through the second feedforward network module to output the feature prediction result. A three-dimensional surface reconstruction method as described in claim 1, wherein the feature prediction result is mapped into the feature dimension using a truncated signed distance function mapper. A three-dimensional surface reconstruction method as described in claim 6, wherein the three-dimensional model is expressed in a truncated signed distance function format. A three-dimensional surface reconstruction method as described in claim 1, wherein the volume feature includes a depth, a texture and a spatial information of the object.

Images