WO2024011426A1 - Point cloud geometry data augmentation method and apparatus, encoding method and apparatus, decoding method and apparatus, and encoding and decoding system - Google Patents

Abstract

A point cloud geometry data augmentation method and apparatus, an encoding method and apparatus, a decoding method and apparatus, and an encoding and decoding system. According to the methods in embodiments of the present disclosure, at an encoding end, pixel down-sampling and feature extraction are performed multiple times on geometry data of an Nth scale to obtain feature data for augmenting geometry data of a (N+1)th-scale point cloud; at a decoding end, after pixel up-sampling and feature induction are performed on the feature data, the output feature data is spliced with the geometry data of the (N+1)th-scale point cloud to obtain augmented geometry data of the (N+1)th-scale point cloud, and then subsequent decoding processing is performed to obtain reconstructed geometry features of a Nth-scale point cloud. According to the embodiments of the present disclosure, point cloud coding performance can be improved, and an auto-encoder model used is convenient to train and use.

Description

A point cloud geometric data enhancement, encoding and decoding method, device and system Technical field

Embodiments of the present disclosure relate to, but are not limited to, point cloud compression technology, and more specifically, to an enhancement method, encoding and decoding method, device and system for point cloud geometric data.

Background technique

Point cloud is a set of discrete points randomly distributed in space that expresses the spatial structure and surface properties of a three-dimensional object or scene. Point cloud is a kind of three-dimensional data, which is a collection of vectors under a three-dimensional coordinate system. These vectors can represent (x, y, z) three-dimensional coordinates, and can also represent attribute information such as color and reflectivity. With the vigorous development of emerging technologies such as augmented reality, virtual reality, autonomous driving and robots, point cloud data has become one of its main data forms due to its concise expression of three-dimensional space. However, the amount of point cloud data is huge, and it is difficult to directly store point cloud data. Data consumes a lot of memory and is not conducive to transmission, so the performance of point cloud compression needs to be continuously improved.

Summary of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

An embodiment of the present disclosure provides a point cloud geometric data enhancement method, which is applied to a point cloud decoder, including:

Parse the code stream to obtain feature data used to enhance the i+1th scale point cloud geometric data;

The feature data is subjected to M _i -1 times of voxel upsampling and feature inference through the partial decoder of the i-th decoder network, and the output feature data is spliced with the geometric data to be enhanced in the i+1-th scale point cloud to obtain the i-th i+1 scale point cloud enhanced geometric data;

Among them, i is an integer greater than or equal to 1, and M _i is an integer greater than or equal to 2.

An embodiment of the present disclosure also provides a method for decoding point cloud geometric data, which is applied to a point cloud decoder, including:

The code stream is parsed, and the geometric data of the N+1th scale point cloud obtained is used as the geometric data to be enhanced. Data enhancement is performed according to the point cloud geometric data enhancement method described in any embodiment of the present disclosure, and the N+1th scale point is obtained. Cloud-enhanced geometric data, N≥1;

The remaining decoders in the N-th decoder network perform voxel upsampling and feature inference on the enhanced geometric data of the N+1 scale point cloud. The output data is then subjected to probability prediction and point cloud clipping to obtain the N-th scale point. Reconstructed geometric data of clouds.

An embodiment of the present disclosure also provides a method for encoding point cloud geometric data, which is applied to a point cloud encoder, including:

Perform N times of voxel downsampling on the geometric data of the first-scale point cloud to obtain the geometric data of the second-scale point cloud to the N+1th scale point cloud, N≥1;

Input the geometric data of the Nth scale point cloud into the Nth encoder network of the Nth autoencoder model for M _N times of voxel downsampling and feature extraction, and output the feature data used to enhance the N+1th scale point cloud geometric data. , M _N ≥ 2;

Entropy encoding is performed on the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network.

An embodiment of the present disclosure also provides a point cloud geometric code stream, wherein the geometric code stream is obtained according to the encoding method of point cloud geometric data described in any embodiment of the present disclosure, including the N+1th scale point cloud The geometric data and the feature data output by the Nth encoder network.

An embodiment of the present disclosure also provides a point cloud geometric data enhancement device, including a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the method described in any embodiment of the present disclosure. The point cloud geometric data enhancement method described above.

An embodiment of the present disclosure also provides a point cloud decoder, including a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the method described in any embodiment of the present disclosure. Decoding method of point cloud geometric data.

An embodiment of the present disclosure also provides a point cloud encoder, including a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the method described in any embodiment of the present disclosure. Encoding method for point cloud geometric data.

An embodiment of the present disclosure also provides a point cloud encoding and decoding system, which includes the point cloud encoder described in any embodiment of the present disclosure, and the point cloud decoder described in any embodiment of the present disclosure.

An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, can implement any implementation of the present disclosure. The point cloud geometric data enhancement method described in the example may be able to implement the decoding method of point cloud geometric information described in any embodiment of the present disclosure, or may be able to implement the encoding method of point cloud geometric information described in any embodiment of the present disclosure. .

An embodiment of the present disclosure also provides a point cloud cropping method, which is applied to a point cloud decoder, including:

Analyze the code stream to obtain the number K of occupied voxels in the point cloud to be cropped;

Determine the occupancy probability of the voxels in the point cloud to be clipped;

Divide the M voxels decomposed from the same voxel in the point cloud to be clipped into one group, set the occupation probability of the m voxels with the highest occupancy probability in each group to 1, and then classify the point cloud to be clipped The occupancy probabilities of all voxels in are sorted, and the K voxels with the highest occupancy probabilities are determined as occupied voxels in the point cloud to be clipped, 1≤m<M<K.

Other aspects will be apparent after reading and understanding the drawings and detailed description.

Figure overview

The drawings are used to provide an understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation of the technical solutions of the present disclosure.

Figure 1 is a flow chart of G-PCC encoding;

Figure 2 is a flow chart of G-PCC decoding;

Figure 3 is a schematic diagram of a method for encoding and decoding point cloud geometric information according to an embodiment of the present disclosure;

Figure 4 is a schematic diagram of a method for encoding and decoding point cloud geometric data according to another embodiment of the present disclosure;

Figure 5 is a flow chart of a method for encoding point cloud geometric data according to an embodiment of the present disclosure;

Figure 6 is a schematic diagram of the network structure of the residual layer according to an embodiment of the present disclosure;

Figure 7 is a schematic diagram of the network structure of an encoder according to an embodiment of the present disclosure;

Figure 8 is a schematic diagram of the network structure of the self-attention layer according to an embodiment of the present disclosure;

Figure 9 is a schematic diagram of the process of obtaining neighborhood context features in the point cloud space from input data by the point cloud neighborhood self-attention layer according to an embodiment of the present disclosure;

Figure 10 is a flow chart of a point cloud geometric data enhancement method according to an embodiment of the present disclosure;

Figure 11 is a flow chart of a method for decoding point cloud geometric data according to an embodiment of the present disclosure;

Figure 12 is a schematic network structure diagram of a probability predictor according to an embodiment of the present disclosure;

Figure 13A is a schematic diagram of the occupied status of voxels in the second scale point cloud according to an embodiment of the present disclosure;

Figure 13B is a schematic diagram of the occupied status of voxels in the third scale point cloud according to an embodiment of the present disclosure;

Figure 13C is a schematic diagram of the occupancy probability of voxels in the second-scale point cloud obtained after probability prediction according to an embodiment of the present disclosure;

Figure 14 is a schematic network structure diagram of a decoder according to an embodiment of the present disclosure;

Figure 15 is a schematic diagram of a point cloud geometric data enhancement device according to an embodiment of the present disclosure.

Elaborate

The present disclosure describes multiple embodiments, but the description is illustrative rather than restrictive, and it will be obvious to a person of ordinary skill in the art that within the scope of the embodiments described in the present disclosure Many more embodiments and implementations are possible.

In the description of the present disclosure, the words "exemplary" or "such as" are used to mean an example, illustration, or explanation. Any embodiment described in this disclosure as "exemplary" or "such as" is not intended to be construed as preferred or advantageous over other embodiments. "And/or" in this article is a description of the relationship between associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and they exist alone. B these three situations. "Plural" means two or more than two. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present disclosure, words such as “first” and “second” are used to distinguish the same or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the number and execution order, and words such as "first" and "second" do not limit the number and execution order.

In describing representative exemplary embodiments, the specification may have presented methods and/or processes as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of steps described herein, the method or process should not be limited to the specific order of steps described. As one of ordinary skill in the art will appreciate, other sequences of steps are possible. Therefore, the specific order of steps set forth in the specification should not be construed as limiting the claims. Furthermore, claims directed to the method and/or process should not be limited to steps performing them in the order written, as those skilled in the art will readily understand that such order may be varied and still remain within the spirit and spirit of the disclosed embodiments. within the range.

Point cloud compression algorithms include geometry-based point cloud compression (G-PCC). Geometric compression in G-PCC is mainly implemented through octree models and/or triangular surface models.

In order to facilitate understanding of the technical solutions provided by the embodiments of the present disclosure, a flow chart of G-PCC encoding and a flow chart of G-PCC decoding are first provided. It should be noted that the flow chart of G-PCC encoding and the flow chart of G-PCC decoding described in the embodiment of the present disclosure are only for the purpose of more clearly illustrating the technical solutions of the embodiment of the present disclosure, and do not constitute a guarantee for the embodiment of the present disclosure. limited. Those skilled in the art know that with the evolution of point cloud compression technology and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to point cloud compression architectures similar to G-PCC. The point cloud compressed by the embodiments of the present disclosure It can be a point cloud in the video, but is not limited to this.

In the point cloud G-PCC encoder framework, the point cloud of the input three-dimensional image model is divided into slices, and each slice is independently encoded.

As shown in the flow chart of G-PCC encoding shown in Figure 1, it is applied to the point cloud encoder. For the point cloud data to be encoded, the point cloud data is first divided into multiple slices through slice division. In each slice, the geometric information and attribute information of the point cloud are encoded separately. In the process of geometric encoding, the geometric information is transformed into coordinates so that all point clouds are contained in a bounding box, and then quantized. The quantization mainly plays the role of scaling. Due to the quantization rounding, part of the point cloud The geometric information is the same, and it can be decided whether to remove duplicate points based on parameters. The process of quantifying and removing duplicate points is also called the voxelization process. Then divide the bounding box into an octree. In the geometric information encoding process based on the octree, the bounding box is divided into eight equal parts into eight sub-cubes, and the non-empty sub-cubes (containing points in the point cloud) continue to be divided into eight equal parts until the leaf structure is obtained. The division stops when the point is a 1x1x1 unit cube, and the points in the leaf nodes are arithmetic encoded to generate a binary geometric bit stream, that is, a geometric code stream. In the process of encoding geometric information based on triangle patch set (triangle soup, trisoup), octree division is also required first, but unlike octree-based geometric information encoding, this trisoup does not need to divide the point cloud step by step. It is divided into a unit cube with a side length of 1x1x1, but is divided into sub-blocks (blocks). The division stops when the side length is W. Based on the surface formed by the distribution of point clouds in each block, twelve links between the surface and the block are obtained. At most twelve intersection points (vertex) generated by the edges, the vertex is arithmetic encoded (surface fitting based on the intersection points), and a binary geometry bit stream (i.e., geometry code stream) is generated. Vertex is also used in the implementation of the geometric reconstruction process, and the reconstructed geometric information is used when encoding the attributes of the point cloud.

During the attribute encoding process, color conversion is performed to convert color information (ie, attribute information) from RGB color space to YUV color space. Then, the point cloud is recolored using the reconstructed geometric information so that the unencoded attribute information corresponds to the reconstructed geometric information. In the process of color information encoding, there are two main transformation methods. One is the distance-based lifting transformation that relies on Level of Detail (LOD) division, and the other is the direct Region Adaptive Hierarchal Transform (Region Adaptive Hierarchal Transform). , RAHT) transformation, both methods will convert the color information from the spatial domain to the frequency domain, obtain high-frequency coefficients and low-frequency coefficients through transformation, and finally quantize the coefficients (i.e., quantization coefficients). Finally, the octree will be passed After the geometric encoding data of division and surface fitting and the attribute encoding data of quantization coefficient processing are slice-synthesized, the vertex coordinates of each block are sequentially encoded (i.e., arithmetic encoding) to generate a binary attribute bit stream, that is, an attribute code stream.

The flow chart of G-PCC decoding shown in Figure 2 is applied to the point cloud decoder. The decoder obtains the binary code stream and independently decodes the geometric bit stream (i.e., the geometric code stream) and the attribute bit stream in the binary code stream. When decoding the geometry bit stream, the geometric information of the point cloud is obtained through arithmetic decoding, octree synthesis, surface fitting, reconstructed geometry and inverse coordinate transformation; when decoding the attribute bit stream, through arithmetic decoding, inverse coordinate transformation Quantization, LOD-based inverse lifting or RAHT-based inverse transformation, and inverse color conversion are used to obtain the attribute information of the point cloud, and the three-dimensional image model of the point cloud data is restored based on the geometric information and attribute information.

Neural network and deep learning technology can also be applied to point cloud geometry compression technology. For example, volume model compression technology based on 3D Convolution Neural Network (3D CNN) directly uses multi-layer perceptron based on point coordinate collection. (Multi-Layer Perceptron, MLP) neural network compression technology, compression technology that uses MLP or 3D CNN for probability estimation and entropy coding of octree node symbols, and compression technology based on three-dimensional sparse convolutional neural networks, etc. wait. Point clouds can be divided into sparse point clouds and dense point clouds according to the density of points. Sparse point clouds have the characteristics of large representation range and sparse distribution in three-dimensional space, and can represent a scene; while dense point clouds have the characteristics of small representation range and sparse distribution. Dense features can represent an object. The compression performance of the above compression technologies on these two point clouds is often quite different, with better performance on dense point clouds and worse performance on sparse point clouds.

In order to improve the performance of the neural network-based encoding and decoding method on sparse point clouds, embodiments of the present disclosure provide a point cloud geometric encoding and decoding method based on an autoencoder model, which can achieve lossy compression of point clouds.

The encoding method of point cloud geometric data in the embodiment of the present disclosure can be applied to the geometric information encoding process of G-PCC as shown in Figure 1, replacing the encoding process after voxelization is completed (such as octree division, surface fitting, etc. ), get the geometric code stream. The decoding method of point cloud geometric data in the embodiment of the present disclosure can be applied to the geometric information decoding process of G-PCC as shown in Figure 2, replacing the decoding processing of the geometric code stream before inverse coordinate transformation (such as octree synthesis, Surface fitting, etc.) to obtain reconstructed geometric data of the point cloud. The entropy coding in the encoding method of the embodiment of the present disclosure can use the arithmetic coding method in Figure 1, and the entropy decoding in the decoding method of the embodiment of the present disclosure can use the arithmetic decoding method in Figure 2. However, the encoding and decoding method of point cloud geometric data in the embodiment of the present disclosure can also be used in other point cloud encoding and decoding processes besides G-PCC.

A schematic diagram of a method for encoding and decoding point cloud geometric data according to an embodiment of the present disclosure is shown in Figure 3. At the encoding end, the geometric data of the first scale point cloud is subjected to voxel downsampling twice. The first-scale point cloud can be the original-scale point cloud to be encoded. After performing voxel downsampling on the geometric data of the first-scale point cloud, the geometric data of the second-scale point cloud is obtained; After another voxel downsampling of the geometric data, the geometric data of the third-scale point cloud is obtained. The geometric data of the third-scale point cloud is entropy-encoded to generate a geometric code stream. The decoding end can obtain lossless geometric data of the third-scale point cloud through entropy decoding, and it is necessary to obtain reconstructed geometric data of higher scales (such as second-scale point cloud, first-scale point cloud) based on the geometric data of the third-scale point cloud. .

In order to improve the accuracy of reconstructed geometric data at higher scales, embodiments of the present disclosure enhance the geometric data of low-scale point clouds through an autoencoder model. Specifically, on the encoding side, this embodiment performs at least two voxel downsampling and feature extraction on the geometric data of the second-scale point cloud through the encoder network to obtain feature data used to enhance the geometric data of the third-scale point cloud. , in the figure, voxel downsampling (step size 2×2×2) and feature extraction are performed through two encoders respectively to extract feature data that is truly helpful for reconstruction and reduce the amount of data to be transmitted. In this article, the feature data extracted through the neural network is called latent feature data. The feature data output by the encoder network is quantized and entropy-encoded and written into the code stream, or it can also be directly entropy-encoded and written into the code stream.

At the decoding end, lossless geometric data of the third-scale point cloud and feature data used to enhance the geometric data of the third-scale point cloud are obtained through entropy decoding. This lossless geometric data is the geometric data to be enhanced in the third scale point cloud. After a decoder of the decoder network performs voxel upsampling and feature inference on the feature data, the output feature data is spliced with the geometric data to be enhanced in the third scale point cloud to obtain the enhanced third scale point cloud. Geometric data, represented in the figure as geometric data + feature data of the third-scale point cloud.

As shown in Figure 3, after obtaining the enhanced geometric data of the third-scale point cloud, voxel upsampling and feature inference are performed on the enhanced geometric data of the third-scale point cloud through another decoder of the decoder network. The data output by the decoder is then subjected to probability prediction and point cloud clipping to obtain reconstructed geometric data of the second-scale point cloud. After the geometric data of the third-scale point cloud is enhanced with features, the reconstructed geometric data of the second-scale point cloud obtained by decoding is closer to the original geometric data of the second-scale point cloud, which can significantly improve the decoding performance. The above-mentioned encoder network and decoder network belong to the same autoencoder model, and the network parameters of the two are obtained through joint training.

The reconstructed geometric data of the second-scale point cloud can continue to be fed into the probabilistic prediction model, which performs voxel upsampling and feature inference, as well as probabilistic prediction and point cloud clipping, to obtain the reconstructed geometric data of the first-scale point cloud. The decoder used here for one-time voxel upsampling and feature inference can adopt the same structure as the decoder in the decoder network, or it can be designed separately.

The illustrated embodiment does not enhance the reconstructed geometric data of the second-scale point cloud. However, in other embodiments, the reconstructed geometric data of the second-scale point cloud can also be enhanced in a similar manner to obtain the second-scale point. The cloud-enhanced geometric data is then fed into the probabilistic prediction model to obtain the reconstructed geometric data of the first-scale point cloud. Whether enhancement is needed can be determined based on the required overhead and the extent of performance improvement, and this disclosure is not limited in this regard.

In the encoding and decoding process of point cloud geometric data, the embodiments of the present disclosure can flexibly enhance the geometric data of point clouds of different scales. The training of the autoencoder model can be implemented based on the geometric data of two adjacent scale point clouds. It is not necessary to design the encoding network and the decoding network for point clouds of all scales and then train them together like other methods. It is simple, convenient and has good portability.

A schematic diagram of a method for encoding and decoding point cloud geometric data according to an embodiment of the present disclosure is shown in Figure 4. In this embodiment, the number of times i of voxel downsampling for the geometric data of the first scale point cloud is greater than or equal to 3 times, which is better than that shown in Figure 3. More examples are shown. The i+1th scale point cloud shown in the figure is the smallest scale point cloud, and the geometric data of this point cloud is losslessly compressed through entropy coding. In addition to enhancing the geometric data of the point cloud at the smallest scale, that is, the i+1th scale, this embodiment also enhances the reconstructed geometric data of the point cloud at the next smallest scale, that is, the i-th scale. The method of enhancing the i+1th scale point cloud geometric data can be found in the method of enhancing the third scale point cloud geometric data in the embodiment shown in Figure 3. The two are the same, only the numbers of the point cloud and the encoding and decoding network are different. The i-th encoder network (including two encoders) and i-th decoder network (including two decoders) used in it both belong to the i-th autoencoder model.

As shown in the figure, in this embodiment, in order to enhance the reconstructed geometric data of the i-th scale point cloud, the i-1th encoder network (including two encoders) is used at the encoding end to perform the processing on the geometric data of the i-1th scale point cloud. Mi _-1 times of voxel downsampling and feature extraction are performed to obtain feature data used to enhance the i-th scale point cloud geometric data. The structures of the i-1th encoder network and the i-th encoder network used here can be the same or different, and can be trained separately. The number of voxel down-sampling and feature extraction operations Mi _-1 for the geometric data of the i-1th scale point cloud and the number of voxel down-sampling and feature extraction operations for the geometric data of the i-th scale point cloud Mi _i are both greater than 2. , can also be the same or different. The feature data used to enhance the i-th scale point cloud geometric data is quantized and entropy-encoded and written into the geometry code stream, or is entropy-encoded and written into the geometry code stream.

At the decoding end, the feature data used to enhance the geometric data of the i-th scale point cloud is obtained through entropy decoding. At the same time, a voxel upsampling and feature inference are performed on the feature-enhanced geometric data of the i+1-th scale point cloud, as well as probability Through prediction and point cloud clipping, the reconstructed geometric data of the i-th scale point cloud can be obtained, which is the geometric data to be enhanced. After a voxel upsampling and feature inference is performed on the feature data used to enhance the i-th scale point cloud geometric data through a decoder of the i-1 decoder network, the output feature data is consistent with the i-th scale point cloud to be enhanced. The geometric data is spliced to obtain the enhanced geometric data of the i-th scale point cloud.

As shown in Figure 4, after obtaining the enhanced geometric data of the i-th scale point cloud, a voxel upsampling and feature inference are performed on the enhanced geometric data of the i-th scale point cloud through the probabilistic prediction model, as well as probability prediction and point Cloud clipping, outputs the reconstructed geometric data of the i-1th scale point cloud. In this probabilistic prediction model, another decoder of the i-1 decoder network is used to perform voxel upsampling and feature inference on the geometric data after the i-th scale point cloud enhancement. The i-1 decoder network and the i-1 encoder network both belong to the i-1 autoencoder model. The i-1th decoder network in the illustrated example includes two encoders, and the i-1th encoder network includes two decoders. However, in other embodiments, more encoders may be used to implement more times of voxel downsampling and feature extraction, and more decoders may be used to implement more times of voxel upsampling and feature inference.

This embodiment enhances the geometric data of the smallest scale point cloud and the sub-small scale point cloud, but this is only exemplary. In other embodiments, the geometric data of more scale point clouds can also be enhanced, or The implementation methods for enhancing the geometric data of the smallest scale point cloud and other scale point clouds except the sub-small scale are similar and will not be described again here. Which scale point cloud geometric data should be enhanced can be determined based on the required overhead and the extent of performance improvement.

Take Figure 3 as an example. When training the autoencoder model, connect the encoder network and decoder network as shown in the figure. However, the entropy encoding and entropy decoding in the figure can be cancelled. You can use the commonly used methods for training deep neural networks. For point cloud samples, the training loss function can be set to BCE (Binary Cross Entropy) loss, which is the difference between the occupation probability of voxels in the third-scale point cloud obtained through probability prediction and the actual occupation symbol of the voxel in the third-scale point cloud. Cross entropy.

Embodiments of the present disclosure also provide a method for encoding point cloud geometric data, as shown in Figure 5, including:

Step 110: Perform N times of voxel downsampling on the geometric data of the first scale point cloud to obtain the geometric data of the second scale point cloud to the N+1th scale point cloud, N≥1;

Step 120: Input the geometric data of the Nth scale point cloud into the Nth encoder network of the Nth autoencoder model for M _N times of voxel downsampling and feature extraction, and the output is used to enhance the N+1th scale point cloud geometric data. Characteristic data, M _N ≥ 2;

Step 130: Entropy encoding is performed on the geometric data of the N+1 scale point cloud and the feature data output by the Nth encoder network.

In the above step 110, before performing voxel downsampling on the geometric data of the first-scale point cloud, the voxelization of the point cloud geometric information needs to be completed. After voxelization, the point cloud is presented as a voxel grid. A voxel is the smallest unit in the voxel grid. A point in the point cloud corresponds to an occupied voxel (i.e., a non-empty voxel), while an unoccupied voxel (i.e., an empty voxel) indicates that there is no point. The geometric data of point clouds can be represented in different ways. For example, the geometric data of the point cloud can be represented by the occupation symbols of the voxels in the point cloud (also called placeholders, placeholders, etc.). The occupied voxels are marked as 1, and the unoccupied voxels are marked as 1. Marked as 0, a binary symbol sequence is obtained. For another example, the geometric data of a point cloud can also be expressed in the form of a sparse tensor, and the coordinate data of all points in the point cloud are arranged in an agreed order. Different representations can be converted to each other.

In the encoding method of point cloud geometric data in the embodiment of the present disclosure, at the encoding end, the geometric data of the Nth scale point cloud is input into the Nth encoder network of the Nth autoencoder model to perform M _N times of voxel downsampling and feature extraction, and output Feature data used to enhance N+1 scale point cloud geometric data. After the feature data is entropy encoded, it is transmitted to the decoder along with the geometric code stream. This feature data is the feature data extracted by taking the geometric data of the Nth scale point cloud as input, which contains the implicit geometric information of the higher scale point cloud that is not covered by the geometric data of the N+1th scale point cloud. Helps the decoder to enhance the geometric data of the N+1 scale point cloud, thereby obtaining more accurate reconstructed geometric data of the Nth scale point cloud. Improve the quality of the reconstructed point cloud, and the feature data has been down-sampled multiple times, requiring less data to be transmitted, which can improve the efficiency of point cloud compression.

In an exemplary embodiment of the present disclosure, when performing voxel downsampling on the geometric data of the first-scale point cloud, it can be implemented through a simple pooling method. For example, a maximum pooling layer with a step size of 2×2×2 is used to merge 8 voxels of the first-scale point cloud into 1 voxel of the second-scale point cloud, thus achieving one voxel downsampling. Sampling reduces the size of the point cloud to half its original size in all three dimensions. Between two scale point clouds, the larger one can be called a high-scale point cloud, and the smaller one can be called a low-scale point cloud. Among the point clouds obtained by N times of voxel downsampling, the N+1th scale point cloud is the point cloud with the smallest scale, has the smallest amount of data, and can be written into the code stream through entropy encoding.

Please refer to Figure 3. The third-scale point cloud in the figure includes 2×2×1 voxels, while the second-scale point cloud includes 4×4×2 voxels, and the first-scale point cloud includes 8×8×4 voxels. Only solid cubes are used in the figure to show the occupied voxels in the point cloud at each scale. The point cloud shown in Figure 3 is only exemplary, and actual point clouds usually include many more voxels. There is a certain degree of correlation between the geometric data of low-scale point clouds and the geometric data of high-scale point clouds. For example, an occupied voxel in a low-scale point cloud is surrounded by occupied voxels (for example, the voxel is (located in the middle of an object), after the voxel is decomposed into multiple voxels in the high-scale point cloud, the multiple voxels obtained by decomposition have a greater probability of being occupied voxels. These correlations can be represented by features extracted by neural networks.

In an exemplary embodiment of the present disclosure, before entropy encoding the feature data, the method further includes: quantizing the feature data. Quantization can reduce the codewords required to transmit feature data, but will also bring certain losses.

In an exemplary embodiment of the present disclosure, the method further includes entropy encoding the number K _N of occupied voxels in the Nth scale point cloud. After the number K _N is written into the geometric code stream through entropy encoding, it can be used for point cloud clipping at the decoder to improve the accuracy of point cloud clipping.

In an exemplary embodiment of the present disclosure, the method further includes:

When N≥2, the geometric data of the j-th scale point cloud is input into the j-th encoder network of the j-th autoencoder model for M _j voxel downsampling and feature extraction, and the output is used to enhance the j+1-th scale point. Cloud geometry data point cloud feature data;

The feature data output by the jth encoder network is quantized and entropy encoded, M _j ≥ 2, and the value of j is any one or more of {1, 2,...,N-1}.

That is, when there are point clouds of more than three scales, this embodiment not only enhances the geometric data of the smallest scale point cloud, but also enhances the geometric data of one or more other scale point clouds except the first scale point cloud. For enhancement, please refer to the encoding process shown in Figure 4. For example, in the case of N=4, there are point clouds in 5 scales from the first scale to the fifth scale. In addition to enhancing the geometric data of the smallest scale, that is, the fifth-scale point cloud, when the value of j can be 3, it means that the geometric data of the fourth-scale point cloud is also enhanced. When the value of j is 2, it means that the geometric data of the fourth-scale point cloud is also enhanced. To enhance the geometric data of the third-scale point cloud, when the value of j is 2 and 3, it means that the geometric data of the third-scale point cloud and the fourth-scale point cloud are also enhanced, and so on. When j values are different, the values of M _j can be the same or different.

The encoder network recorded in this article performs voxel downsampling and feature extraction. It does not mean that the encoder network performs voxel downsampling first and then performs feature extraction. Voxel downsampling can be performed before feature extraction or during feature extraction. It can be performed later or between multiple feature extractions, and this disclosure does not impose any limitations on this. Similarly, the decoder network recorded in this article performs voxel upsampling and feature inference, which does not mean that the decoder network first performs voxel upsampling and then performs feature inference. Voxel upsampling can be performed before feature inference, or it can be performed before feature inference. It can be performed after feature inference or between multiple feature inferences, and this disclosure does not impose any limitations on this.

In an exemplary embodiment of the present disclosure, each time of voxel downsampling and feature extraction includes:

Perform feature extraction on the input data through at least one of a first residual network and a first self-attention network based on sparse convolution;

Perform voxel downsampling on the data output by the first residual network or the first self-attention network through a sparse convolution layer with a stride of 2×2×2;

Feature extraction is performed on the data output by the sparse convolution layer through at least one of a second residual network based on sparse convolution and a second self-attention network.

Each time the encoder network in this embodiment performs voxel downsampling and feature extraction, it does so in the manner of feature extraction, voxel downsampling, and feature extraction.

In an example of this embodiment, the first residual network and the second residual network include one or more residual layers based on sparse convolution. Each residual layer is shown in Figure 6 and includes three For the above branches, branch one directly outputs the input data, and other branches perform feature inference on the input data through different numbers of sparse convolution layers. The outputs of the other branches are spliced and then added to the output of branch one to obtain the residual. The output of the layer. Three branches are shown in Figure 6. Branch two includes two sparse convolution layers, and branch three includes three sparse convolution layers. There are activation functions between adjacent sparse convolution layers.

In an example of this embodiment, each time the voxel downsampling and feature extraction are implemented through an encoder based on a neural network, as shown in Figure 7, the encoder includes: a first sparse convolutional network, a third A self-attention network, a first residual network, a sparse convolution layer with a step size of 2×2×2, a second residual network, a second self-attention network, and a second sparse convolution network; in the There is an activation function between the first sparse convolution network and the first self-attention network, and between the first residual network and the sparse convolution layer. The first sparse convolution network and the Two sparse convolutional networks include one or more sparse convolutional layers.

In an exemplary embodiment of the present disclosure, the first self-attention network and/or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes:

For each point in the point cloud, search for the neighbor points of the point based on the coordinate data of the point, linearly transform the distance information from the point to the neighbor point to obtain the location feature, and compare the location feature with the neighbor point. The features of the points are added to obtain the aggregated features after position encoding;

Perform a first linear transformation on the input feature data to obtain a first vector, perform matrix multiplication of the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and after activation, a point cloud is obtained. The attention weight of each point relative to its neighbor points;

Data containing the neighborhood context features are obtained by multiplying the attention weight and a third vector, and the third vector is obtained by performing a third linear transformation on the aggregated features.

In an example of this embodiment, as shown in Figure 8, the self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. , the point cloud neighborhood self-attention layer is used to obtain neighborhood context features in the point cloud space from the input data. The output data of the point cloud neighborhood self-attention layer is added to the input data and then input to the first normalization layer. Batch normalization is performed, and the results are then input to the linear layer for linear transformation. The output data of the linear layer and the input data are added and then input to the second normalization layer for batch normalization, and the self-attention layer is obtained. output.

和坐标C _in∈R ^n×3，坐标C _in∈R ^n×3用于邻居点的查找。n为点云的总点数，d _in为输入的特征数据的维度。 An example of this embodiment is shown in Figure 9. The point cloud data (input) of the input point cloud neighborhood self-attention layer includes feature data

and the coordinates C _in ∈R ^n×3 , the coordinates C _in ∈R ^n×3 are used to find neighbor points. n is the total number of points in the point cloud, and _din is the dimension of the input feature data.

As shown in the figure, the processing performed by the point cloud neighborhood self-attention layer includes:

K nearest neighbor (KNN) search:

在一些算法中，可以将该点也算成是该点的k个邻居点中的一个，在另一些算法中，该点不作为该点的邻居点，对此本公开不做局限。 For each point p _i in the point cloud, use the K nearest neighbor search algorithm (k nearest neighbor, KNN) to find the k nearest neighbor points {p _i1 , p _i2 ,...p _ik } to the point, and aggregate them to obtain k The coordinates of neighbor points C _knn ∈R ^n×k×3 and features

In some algorithms, this point can also be counted as one of the k neighbor points of this point. In other algorithms, this point is not regarded as a neighbor point of this point, and this disclosure is not limited.

Location code:

Taking each point p _i as the center point, find the relative distance {dist _i1 , dist _i2 ,...dist _ik } between p _i 's k neighbor points {p _i1 , p _i2 ,...p _ik } and the center point p _i , and get Relative distance information dist _knn ∈R ^n×k×1 . Then the dimension of dist _knn is mapped from 1 to _din dimension through the linear layer W _dist , and the obtained relative position feature is added to the feature F _knn (that is, appended to the feature F _knn ) to achieve position encoding:

F′ _knn ＝dist _knn ·W _dist +F _knn

F′ _knn是位置编码后的聚合特征，

通过位置编码，为特征赋予了对应点之间相对位置的感知信息，每个邻居点的特征都具有了空间位置信息。 in,

F′ _knn is the aggregated feature after position encoding,

Through position coding, features are given perceptual information of relative positions between corresponding points, and the features of each neighbor point have spatial position information.

QKV vector generation:

The input feature data F _in is transformed through the linear layer W _Q to obtain the Q vector. The position-encoded aggregate feature F′ _knn is transformed through the linear layer W _K and the linear layer W _V respectively to obtain the K vector and V vector, namely:

Q＝F _in ·W _q ,(K,V)＝F′ _knn ·(W _k ,W _v )

和

表示3个不同的线性变换。Q向量代表查询向量(Query)，K向量代表被查询信息与其他信息的相关性的向量(Key)，V向量代表被查询信息的向量(Value)。上述维度参数d _a和d _out可以等于d _in，如均设置为32。d _a和d _out也可以不等于d _in即可以进行维度变换。 in,

and

Represents 3 different linear transformations. The Q vector represents the query vector (Query), the K vector represents the vector (Key) of the correlation between the queried information and other information, and the V vector represents the vector (Value) of the queried information. The above dimension parameters d _a and d _out can be equal to d _in , for example, both are set to 32. D _a and d _out can also be different from d _in , that is, dimensional transformation can be performed.

Attention weight generation and attention-based feature aggregation:

After obtaining the Q vector, K vector and V vector, perform matrix multiplication of the Q vector and the K vector. The result is activated by the Softmax activation function, and the attention weight A relative to its neighbor point when each point is used as the center point is output. Finally, the attention The force weights A and V vectors are multiplied to obtain the feature data F _out of the output point cloud. Right now:

In this example, before activation, the result of matrix multiplication of Q vector and K vector can also be multiplied by the scaling factor.

Compared with neural networks based solely on sparse convolutions, the above embodiments of the present disclosure can enhance spatial modeling capabilities on sparse point clouds by introducing attention mechanism networks. Because it is difficult for a convolutional network with a fixed convolution kernel size to extract effective neighbor features (i.e., features of neighborhood context) on sparsely distributed point clouds, the above-mentioned embodiments of the present disclosure introduce a network based on an attention mechanism to directly collect points on a point cloud. Based on the k nearest neighbor algorithm, k points around the center point are obtained, and then the attention weight of the center point to other points is obtained through the attention mechanism, which can more effectively extract the feature information of the neighborhood context and improve compression on sparse point clouds. performance.

An embodiment of the present disclosure provides a point cloud geometric data enhancement method, as shown in Figure 10. The method includes:

Step 210, parse the code stream to obtain feature data used to enhance the geometric data of the i+1-th scale point cloud; the feature data is obtained by performing M _i voxels on the geometric data of the i-th scale point cloud through the i-th encoder network Obtained by downsampling and feature extraction, i≥1, M _i≥2 ;

The i-th encoder network in this embodiment can be configured with M _i encoders in cascade, and each encoder performs voxel downsampling and feature extraction on the input data. However, in other embodiments, the number of encoders is variable, and a single encoder can also implement multiple voxel downsampling and feature extraction.

Step 220: Perform Mi _-1 voxel upsampling and feature inference on the feature data through the partial decoder of the i-th decoder network, and the output feature data is spliced with the geometric data to be enhanced in the i+1-th scale point cloud. , obtain the enhanced geometric data of the i+1th scale point cloud;

Wherein, the i-th encoder network and i-th decoder network both belong to the i-th autoencoder model.

In an example of this embodiment, the output feature data includes Li ₊₁ feature data, and the reconstructed geometric data of the i+1th scale point cloud includes coordinate data of Li ₊₁ points; Splicing is to splice the Li ₊₁ feature data and the coordinate data of Li ₊₁ points in one-to-one correspondence to obtain the coordinates and feature data of Li ₊₁ points. Li ₊₁ is the i+1th scale. The number of points in the point cloud. When the encoding end performs voxel downsampling and feature extraction on the geometric data of the i-th scale point cloud, the obtained feature data (such as eigenvalues) and geometric data (such as point coordinates) are in one-to-one correspondence in order. At the decoding end, by splicing the two together, the coordinates and feature data of each point in the point cloud can be obtained. In other words, the characteristic value of each occupied voxel in the point cloud can be obtained.

In an example of this embodiment, the i-th encoder network implements the voxel downsampling through sparse convolution with a step size of 2×2×2; the i-th decoder network implements the voxel downsampling through a step size of 2×2 A 2×2 transposed sparse convolution implements the voxel upsampling. Voxel downsampling is implemented through sparse convolution in the encoder network, and voxel upsampling is implemented through transposed sparse convolution in the decoder network. The parameters of sparse convolution and transposed sparse convolution are both learnable, which is beneficial to Improve compression encoding performance.

An embodiment of the present disclosure also provides a method for decoding point cloud geometric data, as shown in Figure 11, including:

Step 310: Parse the code stream and obtain the geometric data of the N+1th scale point cloud as the geometric data to be enhanced. Enhance it according to the point cloud geometric data enhancement method described in any embodiment of the present disclosure to obtain the N+1th scale point cloud. Scaled point cloud enhanced geometric data, N≥1;

Step 320: Use the remaining decoders of the N-th decoder network to perform voxel upsampling and feature inference on the enhanced geometric data of the N+1-th scale point cloud. The output data is then subjected to probability prediction and point cloud clipping to obtain the N-th scale point cloud. Reconstructed geometric data from N-scale point clouds.

The remaining decoders of the above-mentioned Nth decoder network and the network used to perform probability prediction and point cloud clipping constitute the Nth probability prediction model. The output of the Nth probability prediction model is the reconstructed geometric data of the Nth scale point cloud. Probabilistic prediction can be achieved through probabilistic predictors, and point cloud clipping can be achieved through clippers for point clouds.

The decoding method of point cloud geometric data in the embodiment of the present disclosure uses the decoded feature data to enhance the geometric data of the N+1th scale point cloud at the decoding end, and then performs the processing based on the enhanced geometric data of the N+1th scale point cloud. Voxel upsampling and feature inference, as well as probability prediction and point cloud clipping, are used to obtain the reconstructed geometric data of the Nth scale point cloud. This feature data is obtained by performing M _N times of voxel downsampling and feature extraction on the geometric data of the Nth scale point cloud. It contains the implicit feature information of the Nth scale point cloud, which can help the decoder to obtain a more accurate Nth Reconstructed geometric data from scaled point clouds. Improve the quality of the reconstructed point cloud, and the feature data has been down-sampled multiple times, requiring less data to be transmitted, which can improve the efficiency of point cloud compression.

In an exemplary embodiment of the present disclosure, the method further includes:

When N≥2, the reconstructed geometric data of the Nth scale point cloud is input into the cascaded N-1 probabilistic prediction models, and voxel upsampling, feature inference, and probability prediction are performed in each of the probabilistic prediction models. And point cloud clipping, output the reconstructed geometric data of the corresponding scale point cloud;

The reconstructed geometric data of the first-scale point cloud is obtained from the output of the last probabilistic prediction model.

In this embodiment, when voxel downsampling is performed on the first-scale point cloud more than twice, feature enhancement is only performed on the geometric data at the smallest scale, and the reconstructed geometric data at other scales is not enhanced. By inputting the reconstructed geometric data of the N-th scale into a probabilistic prediction model, the reconstructed geometric data of the N-1 scale point cloud is obtained, and then inputting the reconstructed geometric data of the N-1 scale into a probabilistic prediction model to obtain the N-2-th scale point. The reconstructed geometric data of the cloud is obtained until the reconstructed geometric data of the first-scale point cloud is obtained. This process can be seen in Figure 3 and its related description.

In an exemplary embodiment of the present disclosure, the method further includes:

When N ≥ 2, the reconstructed geometric data of the j-th scale point cloud or the enhanced geometric data of the j-th scale point cloud is input into the j-1th probability prediction model, and a single volume is performed in the j-1th probability prediction model. After pixel upsampling and feature inference, as well as probability prediction and point cloud clipping, the reconstructed geometric data of the j-1th scale point cloud is output, j=2,3,...,N;

Wherein, the enhanced geometric data of the j-th scale point cloud is to use the reconstructed geometric data of the j-th scale point cloud as the geometric data to be enhanced for the j-th scale point cloud, according to the point cloud geometric data enhancement described in any embodiment of the present disclosure. obtained after enhancing the method.

In addition to enhancing the geometric data of the smallest scale point cloud, this embodiment can also enhance the reconstructed geometric data of one or more scale point clouds other than the first scale point cloud. The encoding and decoding process shown in Figure 4 is an example of this embodiment. Please refer to Figure 4 and related descriptions. When the input data is feature-enhanced geometric data, the j-1th probabilistic prediction model should use the remaining decoders in the j-1th decoder network to perform voxel upsampling and feature inference, and the j-th The -1 decoder network belongs to the j-1th encoder network of the same autoencoder model, and is used to perform multiple voxel downsampling and feature extraction on the j-1th scale point cloud to obtain the j-th scale point cloud. Feature data of point cloud geometric data. When the input data is reconstructed geometric data without enhancement, the decoder that performs voxel upsampling and feature inference in the j-1th probabilistic prediction model can be designed separately.

In an exemplary embodiment of the present disclosure, the probability prediction is implemented through multiple sparse convolution layers and a sigmod function. In an example of this embodiment, a probability predictor as shown in Figure 12 can be used to implement probability prediction. The probability predictor includes 3 sparse convolution layers, 2 activation functions (such as ReLU functions) set between adjacent sparse convolution layers, and a Sigmod function set in the last layer. The Sigmod function outputs the points obtained by inference Occupancy probability of voxels in the cloud. The numerical range of occupancy probability can be limited to between 0 and 1. The sparse convolution layer can use SConv K1 ³ , S1 ³ , C32. The convolution kernel size in three dimensions is 1, the stride is 1, and the number of channels is 32.

After obtaining the occupation probability of a point in a certain scale point cloud, a simple binary classification method can be used to determine the occupied voxels in the point cloud. See Figure 3. Figure 13A shows the second scale point cloud in Figure 3. The occupied situation of voxels, and Figure 13B shows the occupied situation of voxels in the third-scale point cloud in Figure 3, and Figure 13C is the occupation probability of voxels in the second-scale point cloud obtained by probability prediction (representing probability of being occupied). When using binary classification, voxels with an occupation probability not less than a set threshold (such as 0.5) can be regarded as occupied voxels, and voxels with an occupation probability less than a set threshold (such as 0.5) can be regarded as unoccupied voxels. Thus the reconstructed geometric data of the point cloud is obtained. However, using the binary classification method for point cloud cropping is sometimes not accurate enough.

In order to improve the accuracy of point cloud cropping. An exemplary embodiment of the present disclosure provides a method of assisting cropping based on the number of points in a point cloud. At the encoding end, the number of points in the point cloud of one or more scales to be cropped is entropy-encoded, and the decoding end uses this number to assist in determining the occupied voxels.

The decoding method in this embodiment also includes: parsing the code stream to obtain the number K _N of occupied voxels in the N-th scale point cloud, where K _N is also the number of points in the N-th scale point cloud; and, realizing the points in the following manner Cloud clipping: Group M voxels decomposed from the same voxel in the Nth scale point cloud obtained after probability prediction into a group, and set the occupation probability of the m voxels with the highest occupancy probability in each group of voxels to 1. Then the occupancy probabilities of all voxels in the Nth scale point cloud are sorted, and the K _N voxels with the highest occupancy probability are determined as occupied voxels of the Nth scale point cloud, 1≤m<M.

In an example, 8 voxels obtained by decomposing the same voxel can be grouped into a group, and the occupancy probability of the 1, 2, or 3 voxels with the highest occupancy probability in each group of voxels is set to 1. When decomposing voxels of low-scale point clouds, unoccupied voxels do not need to be decomposed, so at least one of the eight decomposed voxels is 1. In other examples, M can also be equal to other values such as 64. When M is larger, the value of m can also increase accordingly.

In this embodiment, before unified sorting, the occupancy probability of at least one voxel with the highest occupancy probability in each group of voxels is set to 1, and then the K _N individuals with the highest occupancy probability are selected using the number of points in the point cloud as a constraint. Voxels are occupied voxels, which can significantly improve the accuracy of point cloud clipping.

In an exemplary embodiment of the present disclosure, each of the voxel upsampling and feature inference includes:

perform feature inference on the input data through at least one of a first residual network and a first self-attention network based on sparse convolution;

Perform voxel upsampling on the data output by the first residual network or the first self-attention network through a transposed sparse convolution layer with a stride of 2×2×2;

Feature inference is performed on the data output by the transposed sparse convolution layer through at least one of a second residual network based on sparse convolution and a second self-attention network.

Each time the decoder network in this embodiment performs voxel upsampling and feature extraction, it does so in the manner of feature extraction, voxel upsampling, and feature extraction.

In an example of this embodiment, the first residual network and the second residual network include one or more residual layers based on sparse convolution. Each residual layer can be seen in Figure 6 and includes more than three branch, branch one directly outputs the input data, and other branches perform feature inference on the input data through different numbers of sparse convolution layers. The outputs of the other branches are spliced and then added to the output of branch one to obtain the residual layer. Output. Three branches are shown in Figure 6. Branch two includes two sparse convolution layers, and branch three includes three sparse convolution layers. There are activation functions between adjacent sparse convolution layers.

In an example of this embodiment, each time the voxel upsampling and feature extraction are implemented through a decoder based on a neural network, as shown in Figure 14, the encoder includes: a first sparse convolutional network, a third A self-attention network, a first residual network, a transposed sparse convolution layer with a stride of 2×2×2, a second residual network, a second self-attention network, and a second sparse convolution network; so An activation function is provided between the first sparse convolution network and the first self-attention network, and between the first residual network and the transposed sparse convolution layer. The first sparse convolution The network and the second sparse convolutional network include one or more sparse convolutional layers.

Although the embodiments of the present disclosure and the above-mentioned embodiments provide a structure of a decoder and an encoder, there are various neural networks that can realize feature extraction and feature reasoning, and all of them may be used in the present disclosure. Therefore, the present disclosure is not limited to a specific network structure disclosed herein, and any neural network that can implement feature extraction or feature inference based on sparse convolution can be used. In particular, the network structure that implements feature extraction and feature inference can be the same, which is called feature extraction in the encoding network and feature inference in the decoding network.

In an example of this embodiment, the first self-attention network and/or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes: in the following manner Obtain neighborhood context features in point cloud space from input data:

For each point in the point cloud, search for the neighbor points of the point based on the coordinate data of the point, linearly transform the distance information from the point to the neighbor point to obtain the location feature, and compare the location feature with the neighbor point. The features of the points are added to obtain the aggregated features after position encoding;

Perform a first linear transformation on the input feature data to obtain a first vector, perform matrix multiplication of the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and after activation, a point cloud is obtained. The attention weight of each point relative to its neighbor points;

Data containing the neighborhood context features are obtained by multiplying the attention weight and a third vector, and the third vector is obtained by performing a third linear transformation on the aggregated features.

In one example, please refer to Figure 8. The self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. The point cloud neighborhood The self-attention layer is used to obtain the neighborhood context features in the point cloud space from the input data. The output data of the point cloud neighborhood self-attention layer and the input data are added and then input to the first normalization layer for batch processing. Normalization, the results are then input to the linear layer for linear transformation, the output data of the linear layer and the input data are added and then input to the second normalization layer for batch normalization, and the self- The output of the attention layer. Among them, the process of obtaining the neighborhood context features in the point cloud space from the input data by the point cloud neighborhood self-attention layer can be seen in Figure 9 and related explanations, and will not be described again here.

The point cloud encoding and decoding methods provided by some embodiments of the present disclosure can realize point cloud geometric lossy compression. By combining the attention mechanism with the convolutional neural network to build an autoencoder model and a probabilistic prediction model, the attention mechanism improves the model's ability to extract features and improves the model's compression performance compared to the existing convolution-based structure.

Some embodiments of the present disclosure propose a probability-based point cloud clipping method for the local density of point clouds, which can improve the model's ability to restore the local density of point clouds.

The encoding and decoding methods of the above embodiments of the present disclosure can be used between point clouds of multiple scales, and the compression of each scale is independent of each other. Scale-scalable encoding can be achieved with high flexibility.

The embodiment of the present disclosure compares the point cloud encoding and decoding method that implements point cloud geometric lossy compression with the G-PCC point cloud compression scheme, and the comparison index is BD-rate. The result is as follows:

BD-rate gain relative to GPCC Embodiments of the present disclosure Arco_Valentino_Dense_vox12 -25% Egyptian_mask_vox12 -16% Facade_00009_vox12 -61% House_without_roof_00057_vox12 -56% Shiva_00035_vox12 -44% Staue_Klimt_vox12 -43% Average -41%

"Arco_Valentino_Dense_vox12" in the table is the 12bit point cloud data provided in the GPCC public test conditions. As can be seen from the above table, compared with the G-PCC point cloud compression scheme, the method of the embodiment of the present disclosure shows certain advantages at each code rate point. Compared with MPEG G-PCC, the average BD-rate is increased by 41%. Compared with MPEG G-PCC method achieves better compression performance.

An embodiment of the present disclosure also provides a point cloud geometric code stream, wherein the geometric code stream is obtained according to the encoding method of point cloud geometric data described in any embodiment of the present disclosure, including the N+1th scale point cloud The geometric data and the feature data output by the Nth encoder network.

An embodiment of the present disclosure also provides a point cloud geometric data enhancement device. As shown in Figure 15, it includes a processor 5 and a memory 6 storing a computer program. When the processor 5 executes the computer program, it can Implement the point cloud geometric data enhancement method described in any embodiment of the present disclosure.

An embodiment of the present disclosure also provides a point cloud decoder, see Figure 15, which includes a processor and a memory storing a computer program. When the processor executes the computer program, it can implement any implementation of the present disclosure. The decoding method of point cloud geometric data described in the example.

An embodiment of the present disclosure also provides a point cloud encoder, see Figure 15, which includes a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement any implementation of the present disclosure. The encoding method of point cloud geometric data described in the example.

An embodiment of the present disclosure also provides a point cloud encoding and decoding system, which includes a point cloud encoder as described in any embodiment of the present disclosure, and a point cloud decoder as described in any embodiment of the present disclosure.

The processor in the above embodiments of the present disclosure may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), a microprocessor, etc., or it may be other conventional processors. Processor, etc.; the processor may also be a digital signal processor (DSP), application specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), discrete logic or other programmable logic devices, discrete gates or transistor logic devices , discrete hardware components; it can also be a combination of the above devices. That is, the processor in the above embodiments can be any processing device or device combination that implements the methods, steps and logical block diagrams disclosed in the embodiments of the present invention. If embodiments of the present disclosure are implemented in part in software, instructions for the software may be stored in a suitable non-volatile computer-readable storage medium and may be executed in hardware using one or more processors. Instructions are provided to perform the methods of embodiments of the present disclosure.

The devices and systems of the above embodiments of the present disclosure can be implemented based on computing devices such as terminals or servers. Terminals may include mobile phones, tablets, laptops, PDAs, Personal Digital Assistants (PDAs), Portable Media Players (PMPs), navigation devices, wearable devices, and smart bracelets. , mobile terminals such as pedometers, and fixed terminals such as digital TVs and desktop computers.

An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it can implement any of the aspects of the present disclosure. The point cloud geometric data enhancement method described in the embodiment may be able to implement the decoding method of point cloud geometric information as described in any embodiment of the present disclosure, or may be able to implement the point cloud geometric information as described in any embodiment of the present disclosure. encoding method.

An embodiment of the present disclosure also provides a point cloud cropping method, which is applied to a point cloud decoder, including:

Analyze the code stream to obtain the number K of occupied voxels in the point cloud to be cropped;

Determine the occupancy probability of the voxels in the point cloud to be clipped;

Divide the M voxels decomposed from the same voxel in the point cloud to be clipped into one group, set the occupation probability of the m voxels with the highest occupancy probability in each group to 1, and then classify the point cloud to be clipped The occupancy probabilities of all voxels in are sorted, and the K voxels with the highest occupancy probabilities are determined as occupied voxels in the point cloud to be clipped, 1≤m<M<K.

In an example of this embodiment, m=1 or 2 or 3, M=8. However, the present disclosure is not limited to this. For example, M may also be 64. The larger M is, the larger m may also be set.

This embodiment not only obtains the accurate number K of occupied voxels in the point cloud to be cropped through decoding, but also divides the M voxels obtained by decomposing the same voxel into one group when sorting the probability, and divides the occupied voxels in each group into The occupation probability of the m voxels with the highest probability is set to 1. Because unoccupied voxels are not subject to probability prediction, and at least one of the decomposed voxels is occupied, the method implemented in this implementation takes advantage of the law of point cloud decomposition and can significantly improve point cloud clipping (i.e., determine point cloud the accuracy of occupied voxels in .

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media that corresponds to tangible media, such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, such as according to a communications protocol. In this manner, computer-readable media generally may correspond to non-transitory, tangible computer-readable storage media or communication media such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory or may be used to store instructions or data. Any other medium that stores the desired program code in the form of a structure and that can be accessed by a computer. Furthermore, any connection is also termed a computer-readable medium if, for example, a connection is sent from a website, server, or using any of the following: coaxial cable, fiber-optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, and microwave or other remote source transmits instructions, then coaxial cable, fiber optic cable, twin-wire, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient (transitory) media, but are directed to non-transitory tangible storage media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, or Blu-ray discs. Disks usually reproduce data magnetically, while optical discs use lasers to reproduce data. Regenerate data optically. Combinations of the above should also be included within the scope of computer-readable media.

May be performed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Execute instructions. Accordingly, the term "processor" as used herein may refer to any of the structures described above or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Furthermore, the techniques may be implemented entirely in one or more circuits or logic elements.

Technical solutions of embodiments of the present disclosure may be implemented in a wide variety of devices or equipment, including wireless handsets, integrated circuits (ICs), or a set of ICs (eg, chipsets). Various components, modules or units are depicted in embodiments of the present disclosure to emphasize functional aspects of devices configured to perform the described techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperating hardware units (including one or more processors as described above) in conjunction with suitable software and/or firmware.

Claims (30)

A point cloud geometric data enhancement method applied to point cloud decoders, including: Parse the code stream to obtain feature data used to enhance the i+1th scale point cloud geometric data; The feature data is subjected to M _i -1 times of voxel upsampling and feature inference through the partial decoder of the i-th decoder network, and the output feature data is spliced with the geometric data to be enhanced in the i+1-th scale point cloud to obtain the i-th i+1 scale point cloud enhanced geometric data; Among them, i is an integer greater than or equal to 1, and M _i is an integer greater than or equal to 2. The method of claim 1, wherein: The feature data is obtained by performing M _i times of voxel downsampling and feature extraction on the geometric data of the i-th scale point cloud through the i-th encoder network; The i-th encoder network and the i-th decoder network both belong to the i-th autoencoder model. The method of claim 1, wherein: The output feature data includes Li ₊₁ feature data, and the reconstructed geometric data of the i+1-th scale point cloud includes coordinate data of Li ₊₁ points; The splicing is to splice the Li ₊₁ feature data and the coordinate data of Li ₊₁ points in one-to-one correspondence to obtain the coordinates and feature data of Li ₊₁ points, where Li ₊₁ is the i+th The number of points in the 1-scale point cloud. A decoding method for point cloud geometric data, applied to point cloud decoders, including: The code stream is parsed, and the geometric data of the N+1th scale point cloud obtained is used as the geometric data to be enhanced. The point cloud geometric data is enhanced according to the method as described in any one of claims 1 to 3 to obtain the N+1th scale. Point cloud enhanced geometric data, N is an integer greater than or equal to 1; The remaining decoders in the Nth decoder network perform voxel upsampling and feature inference on the enhanced geometric data of the N+1 scale point cloud. The output data is then subjected to probability prediction and point cloud clipping to obtain the Nth Reconstructed geometric data from scaled point clouds. The method of claim 4, further comprising: When N≥2, input the reconstructed geometric data of the N-th scale point cloud into a cascade of N-1 probabilistic prediction models, perform voxel upsampling and feature inference in each of the probabilistic prediction models, and Probabilistic prediction and point cloud clipping, output reconstructed geometric data of corresponding scale point cloud; The reconstructed geometric data of the first-scale point cloud is obtained from the output of the last probabilistic prediction model. The method of claim 4, further comprising: When N≥2, the reconstructed geometric data of the j-th scale point cloud or the enhanced geometric data of the j-th scale point cloud is input into the j-1th probability prediction model, and a voxel calculation is performed in the j-th probability prediction model. After sampling and feature inference, as well as probability prediction and point cloud clipping, the reconstructed geometric data of the j-1th scale point cloud is output, j=2,3,...,N; Wherein, the enhanced geometric data of the j-th scale point cloud is to use the reconstructed geometric data of the j-th scale point cloud as the geometric data to be enhanced for the j-th scale point cloud, and is performed according to the method as described in any one of claims 1 to 3. Point cloud geometric data obtained after enhancement. The method of claim 4, wherein: The method further includes: parsing the code stream to obtain the number K _N of occupied voxels in the Nth scale point cloud; The point cloud clipping is implemented in the following way: M voxels decomposed from the same voxel in the Nth scale point cloud obtained after probability prediction are divided into a group, and the m voxels with the highest probability in each group of voxels are grouped. Set the occupancy probability to 1, then sort the occupancy probabilities of all voxels in the Nth scale point cloud, and determine the K _N voxels with the highest occupancy probability as the occupied voxels of the Nth scale point cloud, 1≤m<M . The method of any one of claims 4 to 6, wherein: Each time the voxel upsampling and feature inference includes: perform feature inference on the input data through at least one of a first residual network based on sparse convolution and a first self-attention network; Perform voxel upsampling on the data output by the first residual network or the first self-attention network through a transposed sparse convolution layer with a stride of 2×2×2; Feature inference is performed on the data output by the transposed sparse convolution layer through at least one of a second residual network based on sparse convolution and a second self-attention network. The method of claim 8, wherein: The first residual network and the second residual network include one or more residual layers based on sparse convolution. Each residual layer includes more than three branches. Branch one directly outputs the input data, and the other branches pass through Different numbers of sparse convolution layers perform feature inference on the input data. The outputs of the other branches are spliced and then added to the output of branch one to obtain the output of the residual layer. The method of claim 8, wherein: The first self-attention network and/or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes: obtaining the point cloud space from the input data in the following manner. Neighborhood context features: For each point in the point cloud, search for the neighbor points of the point based on the coordinate data of the point, linearly transform the distance information from the point to the neighbor point to obtain the location feature, and compare the location feature with the neighbor point. The features of the points are added to obtain the aggregated features after position encoding; Perform a first linear transformation on the input feature data to obtain a first vector, perform matrix multiplication of the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and after activation, a point cloud is obtained. The attention weight of each point relative to its neighbor points; Data containing the neighborhood context features are obtained by multiplying the attention weight and a third vector, and the third vector is obtained by performing a third linear transformation on the aggregated features. The method of claim 10, wherein: The self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. The point cloud neighborhood self-attention layer is used to extract data from the input data. The neighborhood context features in the point cloud space are obtained. The output data and input data of the point cloud neighborhood self-attention layer are added together and then input to the first normalization layer for batch normalization, and the results are then input to The linear layer performs linear transformation. The output data of the linear layer and the input data are added and then input to the second normalization layer for batch normalization to obtain the output of the self-attention layer. The method of claim 8, wherein: Each time the voxel upsampling and feature inference are implemented through a decoder based on a neural network, the decoder sequentially includes: a first sparse convolution network, a first self-attention network, a first residual network, a step size It is a 2×2×2 transposed sparse convolution layer, a second residual network, a second self-attention network, and a second sparse convolution network; the first sparse convolution network and the first self-attention An activation function is provided between the force network and between the first residual network and the transposed sparse convolution layer. The first sparse convolution network and the second sparse convolution network include one or more sparse convolutional layers. Convolutional layer. The method of claim 4, wherein: The probability prediction is implemented through multiple sparse convolution layers and sigmod function. An encoding method for point cloud geometric data, applied to point cloud encoders, including: Perform N times of voxel downsampling on the geometric data of the first-scale point cloud to obtain the geometric data of the second-scale point cloud to the N+1th scale point cloud, N≥1; Input the geometric data of the Nth scale point cloud into the Nth encoder network of the Nth autoencoder model for M _N times of voxel downsampling and feature extraction, and output the feature data used to enhance the N+1th scale point cloud geometric data. , M _N ≥ 2; Entropy encoding is performed on the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network. The method of claim 14, wherein: Before performing entropy encoding on the feature data, the method further includes: quantizing the feature data. The method of claim 14, wherein: The method further includes entropy encoding the number K _N of occupied voxels in the Nth scale point cloud. The method of claim 14, wherein the method further includes: When N≥2, the geometric data of the j-th scale point cloud is input into the j-th encoder network of the j-th autoencoder model for M _j voxel downsampling and feature extraction, and the output is used to enhance the j+1-th scale point. Cloud geometry data point cloud feature data; The feature data output by the jth encoder network is quantized and entropy encoded, M _j ≥ 2, and the value of j is any one or more of {1, 2,...,N-1}. The method of claim 14 or 17, wherein: Each time the voxel downsampling and feature extraction includes: Perform feature extraction on the input data through at least one of a first residual network and a first self-attention network based on sparse convolution; Perform voxel downsampling on the data output by the first residual network or the first self-attention network through a sparse convolution layer with a stride of 2×2×2; Feature extraction is performed on the data output by the sparse convolution layer through at least one of a second residual network and a second self-attention network based on sparse convolution. The method of claim 18, wherein: The first residual network and the second residual network include one or more residual layers based on sparse convolution. Each residual layer includes more than three branches. Branch one directly outputs the input data, and the other branches pass through Different numbers of sparse convolution layers perform feature inference on the input data. The outputs of the other branches are spliced and then added to the output of branch one to obtain the output of the residual layer. The method of claim 18, wherein: The first self-attention network and/or the second self-attention network includes one or more self-attention layers, and the processing performed by each self-attention layer includes: obtaining the point cloud space from the input data in the following manner. Neighborhood context features: For each point in the point cloud, search for the neighbor points of the point based on the coordinate data of the point, linearly transform the distance information from the point to the neighbor point to obtain the location feature, and compare the location feature with the neighbor point. The features of the points are added to obtain the aggregated features after position encoding; Perform a first linear transformation on the input feature data to obtain a first vector, perform matrix multiplication of the first vector and a second vector obtained by performing a second linear transformation on the aggregated features, and after activation, a point cloud is obtained. The attention weight of each point relative to its neighbor points; Data containing the neighborhood context features are obtained by multiplying the attention weight and a third vector, and the third vector is obtained by performing a third linear transformation on the aggregated features. The method of claim 20, wherein: The self-attention layer includes a point cloud neighborhood self-attention layer, a first normalization layer, a linear layer and a second normalization layer connected in sequence. The point cloud neighborhood self-attention layer is used to extract data from the input data. The neighborhood context features in the point cloud space are obtained. The output data and input data of the point cloud neighborhood self-attention layer are added together and then input to the first normalization layer for batch normalization, and the results are then input to The linear layer performs linear transformation. The output data of the linear layer and the input data are added and then input to the second normalization layer for batch normalization to obtain the output of the self-attention layer. The method of claim 18, wherein: Each time the voxel downsampling and feature extraction are implemented through an encoder based on a neural network, the encoder includes in turn: a first sparse convolutional network, a first self-attention network, a first residual network, a step size is a 2×2×2 sparse convolution layer, a second residual network, a second self-attention network, and a second sparse convolution network; in the first sparse convolution network and the first self-attention An activation function is provided between the networks and between the first residual network and the sparse convolution layer. The first sparse convolution network and the second sparse convolution network include one or more sparse convolution layers. . A point cloud geometric code stream, wherein the geometric code stream is obtained according to the encoding method of point cloud geometric data as described in any one of claims 14 to 22, including the geometric data of the N+1th scale point cloud and the feature data output by the Nth encoder network. A device for enhancing point cloud geometric data, including a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can realize the point cloud geometric data as claimed in any one of claims 1 to 3 Enhancement method. A point cloud decoder, including a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can realize the decoding of the point cloud geometric data according to any one of claims 4 to 13 method. A point cloud encoder, including a processor and a memory storing a computer program, wherein when the processor executes the computer program, it can implement the encoding of point cloud geometric data as claimed in any one of claims 14 to 22 method. A point cloud encoding and decoding system, which includes the point cloud encoder as claimed in claim 26 and the point cloud decoder as claimed in claim 25. A non-transitory computer-readable storage medium, the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it can achieve the points described in any one of claims 1 to 3 Cloud geometric data enhancement method, or can realize the decoding method of point cloud geometric data as described in any one of claims 4 to 13, or can realize the encoding of point cloud geometric data as described in any one of claims 14 to 22 method. A point cloud cropping method applied to point cloud decoders, including: Analyze the code stream to obtain the number K of occupied voxels in the point cloud to be cropped; Determine the occupancy probability of the voxels in the point cloud to be clipped; Divide the M voxels decomposed from the same voxel in the point cloud to be clipped into one group, set the occupation probability of the m voxels with the highest occupancy probability in each group to 1, and then classify the point cloud to be clipped The occupancy probabilities of all voxels in are sorted, and the K voxels with the highest occupancy probabilities are determined as occupied voxels in the point cloud to be clipped, 1≤m<M<K. The point cloud clipping method according to claim 29, wherein m=1 or 2 or 3, and M=8.

Images