WO2025009366A1 - Information processing device and method - Google Patents

Abstract

The present disclosure relates to an information processing device and method which make it possible to suppress a reduction in coding efficiency. The method includes: using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature quantity set, which is a set of feature quantities of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space, to derive a guide surface feature quantity, which is a feature quantity based on a positional relationship between the guide surface and the points; performing a predetermined calculation using the first coordinate set and the first feature quantity set reflecting the guide surface feature quantity to derive a second coordinate set and a second feature quantity set; encoding the derived second coordinate set; and encoding the derived second feature quantity set. The present disclosure is applicable, for example, to information processing devices, electronic instruments, information processing methods and programs.

Description

Information processing device and method

This disclosure relates to an information processing device and method, and in particular to an information processing device and method that can suppress a decrease in coding efficiency.

Recently, an AI (Artificial Intelligence)-based point cloud compression method (hereinafter referred to as AI-PCC (Artificial Intelligence - Point Cloud Compression)) has been developed, which is a point cloud compression method that has a neural network in the encoder or decoder. For example, the method described in Non-Patent Document 1 is known as one such AI-PCC method. This method uses a sparse 3D-CNN (3-dimensional Convolutional Neural Network) (a type of neural network) in the encoder and decoder.

Jianqiang Wang, Dandan Ding, Zhu Li, and Zhan Ma, “Multiscale Point Cloud Geometry Compression”, 2021 Data Compression Conference (DCC), 2021 Data Compression Conference (DCC), https://ieeexplore.ieee.org/document/9418789 /figures#figures

However, the method described in Non-Patent Document 1 is not always the best method, and other methods are needed. In other words, there is a risk that the coding efficiency will decrease with the conventional methods.

This disclosure has been made in light of these circumstances, and makes it possible to suppress the reduction in coding efficiency.

　An information processing device according to one aspect of the present technology includes a guide surface feature derivation unit that uses a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set; a coordinate encoding unit that encodes the derived second coordinate set; and a feature encoding unit that encodes the derived second feature set, and the information processing device has a code size smaller than the code size of the encoded data of the first coordinate set and the first feature set.

An information processing method according to one aspect of the present technology uses a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points, perform a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, derive a second coordinate set and a second feature set, encode the derived second coordinate set, encode the derived second feature set, and the amount of code for the encoded data of the second coordinate set and the second feature set is less than the amount of code for the encoded data of the first coordinate set and the first feature set.

An information processing device according to another aspect of the present technology is an information processing device including: a coordinate decoding unit that decodes encoded data to generate a first coordinate set, which is a set of coordinates of points that form a point cloud; a feature decoding unit that decodes encoded data to generate a first feature set, which is a set of features of the coordinates; a guide surface feature derivation unit that uses the generated first coordinate set, the generated first feature set, and a guide surface that is a reference surface formed in three-dimensional space to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; and a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set.

An information processing method according to another aspect of the present technology is an information processing method that decodes encoded data to generate a first coordinate set, which is a set of coordinates of points forming a point cloud, decodes the encoded data to generate a first feature set, which is a set of features of the coordinates, derives guide surface features, which are features based on the positional relationship between the guide surface and the points, using the generated first coordinate set, the generated first feature set, and a guide surface, which is a reference surface formed in three-dimensional space, and performs a calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set.

In an information processing device and method according to one aspect of the present technology, a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of those coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, are used to derive guide surface features, which are features based on the positional relationship between the guide surface and the points, and an operation is performed using the first coordinate set and the first feature set reflecting the guide surface features to derive a second coordinate set and a second feature set, the derived second coordinate set is encoded, and the derived second feature set is encoded.

In the information processing device and method of another aspect of the present technology, encoded data is decoded to generate a first coordinate set which is a set of coordinates of points forming a point cloud, the encoded data is decoded to generate a first feature set which is a set of features of the coordinates, the generated first coordinate set, the generated first feature set, and a guide surface which is a reference surface formed in three-dimensional space are used to derive guide surface features which are features based on the positional relationship between the guide surface and the points, and a predetermined calculation is performed using the first coordinate set and the first feature set reflecting the guide surface features to derive a second coordinate set and a second feature set.

FIG. 1 is a diagram illustrating an example of a main configuration of a conventional calculation unit. FIG. 1 is a diagram illustrating an example of a main configuration of a conventional encoding device. FIG. 1 is a diagram illustrating an example of a main configuration of a conventional encoding device. FIG. 1 is a diagram illustrating an example of an encoding/decoding method to which the present technology is applied. FIG. 2 is a diagram illustrating an example of a main configuration of a calculation unit. FIG. 13 is a diagram showing an example of a guide surface. FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. 11 is a flowchart illustrating an example of the flow of an encoding process. 9 is a flowchart continuing from FIG. 8 , illustrating an example of the flow of the encoding process. 13 is a flowchart illustrating an example of the flow of a guide surface feature amount set derivation process. FIG. 2 is a block diagram showing an example of the main configuration of a decoding device. 13 is a flowchart illustrating an example of the flow of a decoding process. 13 is a flowchart continuing from FIG. 12, illustrating an example of the flow of a decoding process. 13A and 13B are diagrams illustrating examples of guide surface feature amounts. FIG. 13 is a diagram showing an example of fitting. FIG. 2 is a block diagram showing an example of the main configuration of a decoding device. 13 is a flowchart illustrating an example of the flow of a decoding process. FIG. 1 is a diagram showing an example of Trisoup. FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. 11 is a flowchart illustrating an example of the flow of an encoding process. 9 is a flowchart continuing from FIG. 8 , illustrating an example of the flow of the encoding process. FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. 11 is a flowchart illustrating an example of the flow of an encoding process. FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. 11 is a flowchart illustrating an example of the flow of an encoding process. 9 is a flowchart continuing from FIG. 8 , illustrating an example of the flow of the encoding process. 13 is a flowchart illustrating an example of the flow of a guide surface feature amount set derivation process. FIG. 2 is a block diagram showing an example of the main configuration of a decoding device. 13 is a flowchart illustrating an example of the flow of a decoding process. 30 is a flowchart continuing from FIG. 29 , illustrating an example of the flow of a decoding process. FIG. 13 is a diagram illustrating an example of a simulation result. FIG. 2 is a block diagram showing an example of a main configuration of a computer.

Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order.
1. Literature supporting technical content and technical terminology 2. Encoding and decoding of point clouds 3. Encoding and decoding of point clouds using guide surfaces 4. Supplementary notes

<1. Literature supporting technical content and technical terminology>
The scope of disclosure of the present technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing the application, and the contents of other documents referenced in the following non-patent documents.

Non-patent document 1: (mentioned above)
Non-patent document 2: Michael Kazhdan, Matthew Bolitho, Hugues Hoppe, "Poisson Surface Reconstruction", Eurographics Symposium on Geometry Processing (2006), Konrad Polthier, Alla Sheffer (Editors)
Non-patent document 3: Stoll Carsten, Karni Zachi, Rossl Christian, Hitoshi Yamauchi, Seidel Hans-Peter, "Template Deformation for Point Cloud Fitting",
Non-patent document 4: Danhang Tang, Philip A. Chou, Christian Hane, Mingsong Dou, Sean Fanello, Jonathan Taylor, Philip Davidson, Onur G. Guleryuz, Yinda Zhang, Shahram Izadi, Andrea Tagliasacchi, Sofien Bouaziz, Cem Keskin, "Deep "Implicit Volume Compression", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1293-1303

In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

<2. Encoding and Decoding Point Cloud>
<Point Cloud>
Conventionally, as 3D data representing the three-dimensional structure of a three-dimensional structure (object of a three-dimensional shape), there exists a point cloud (also called a point group) that represents the object as a collection of many points. Point cloud data (also called point cloud data) is composed of the geometry (position information) and attributes (attribute information) of each point that constitutes the point cloud. The geometry indicates the position of the point in three-dimensional space. The attributes indicate the attributes of the point. The attributes can include any information. For example, the attributes may include color information, reflectance information, normal information, etc. of each point. In this way, the point cloud has a relatively simple data structure and can represent any three-dimensional structure with sufficient accuracy by using a sufficient number of points.

＜AI-PCC＞
Recently, an AI-based point cloud compression method (hereinafter also referred to as AI-PCC (Artificial Intelligence - Point Cloud Compression)) has been developed, which is a point cloud compression method having a neural network in the encoder or decoder. For example, the method described in Non-Patent Document 1 is known as one such AI-PCC method. In this specification, this method is also referred to as a "conventional AI-PCC method." This conventional AI-PCC method uses a sparse 3D-CNN (3-dimensional Convolutional Neural Network) (a type of neural network) in the encoder and decoder. For example, in the case of this conventional AI-PCC method, as shown in FIG. 1, a CNN (Convolution Neural Network) 11 that converts an input sparse tensor into an output sparse tensor is used. In the case of other general AI-PCC methods, some kind of calculation unit 21 that converts an input coordinate set and an input feature set into an output coordinate set and an output feature set, such as a 3D-CNN, a multilayer perceptron, or another neural network, is used.

Figure 2 shows an example of the main configuration of an encoder for this conventional AI-PCC method. As shown in Figure 2, in this encoder 50, a sparse tensor construction unit 51 constructs an input sparse tensor X = {C_X, F_X} from an input point group C_X. Dummy features consisting of a series of 1s are used for the input feature set F_X. CNN 52 (CNN_E1), CNN 53 (CNN_E2), CNN 54 (CNN_E3), CNN 55 (CNN_E4), CNN 56 (CNN_E5), and CNN 57 (CNN_E6) receive the sparse tensor and return the converted sparse tensor. A point encoding unit 58 encodes the points of C_X, the points of the coordinate set obtained by downscaling C_X by 1/2, and the points of the coordinate set obtained by downscaling C_X by 1/4, and outputs a score bit stream. The coordinate encoding unit 59 encodes (Octree encodes) the coordinate set C_Y, which is obtained by downscaling the input point group C_X by 1/8, and outputs a coordinate bit stream. The feature encoding unit 60 encodes the feature set F_Y associated with each coordinate in the coordinate set C_Y, and outputs a feature bit stream.

The main configuration of a decoder for this conventional AI-PCC method is shown in Figure 3. In this decoder 70, the score decoding unit 71 decodes the score bit stream. The coordinate decoding unit 72 is a G-PCC decoder that performs lossless decoding of the coordinate set bit stream and generates (restores) the coordinate set C_Y. The feature decoding unit 73 entropy decodes the feature bit stream and generates (restores) the feature set F_Y. However, the feature bit stream is lossy compressed, and strictly speaking, the restored feature set F_Y may differ from F_Y before encoding. CNN74 (CNN_D1), CNN75 (CNN_D2), CNN77 (CNN_D3), CNN78 (CNN_D4), CNN80 (CNN_D5), and CNN81 (CNN_D6) are sparse 3D-CNNs that receive sparse tensors and return sparse tensors after transformation. These are mainly composed of sparse 3D convolution layers and nonlinear transformation layers. The occupancy state classification unit 76, the occupancy state classification unit 79, and the occupancy state classification unit 82 are each composed of a single sparse 3D convolution layer. This sparse 3D convolution layer inputs a sparse tensor and outputs a sparse tensor as a result of the operation. The feature of the sparse tensor as a result of the operation of this sparse 3D convolution layer represents the occupancy probability value of the coordinate. The occupancy state classification unit 76, the occupancy state classification unit 79, and the occupancy state classification unit 82 leave only the top k coordinates with the highest occupancy probability value and delete the other coordinates. In other words, the occupancy state classification unit 76, the occupancy state classification unit 79, and the occupancy state classification unit 82 output a coordinate set in which the coordinate set of the input sparse tensor is deleted except for the top k coordinates, and a feature set corresponding to the coordinate set (i.e., the feature set of the input sparse tensor in which the feature corresponding to the deleted coordinates has been deleted). k uses the corresponding scale among N_X, N_X', and N_X''. The feature corresponding to the deleted coordinate in F^out is also deleted.

However, the method described in Non-Patent Document 1 is not always the best method, and other methods were needed. In other words, there was a risk that the coding efficiency would decrease with the conventional methods.

<3. Encoding and decoding of point clouds using guide surfaces>
<Method 1>
Therefore, as shown in the top row of the table in Fig. 4, a coordinate set and a feature set derived using the guide surface feature are transmitted (Method 1). That is, on the transmitting side, the coordinate set and the feature set derived using the guide surface feature are encoded and a bit stream is transmitted, and on the receiving side, the bit stream is received and decoded.

For example, in the case of a calculator 100 using CNN as in the conventional AI-PCC method described above, as shown in FIG. 5, the guide surface feature derivation unit 101 reflects the set of guide surface features g(C^in) in the input sparse tensor {C^in, F^in}, and inputs the input sparse tensor {C^in, [F^in, g(C^in)]} to the CNN 102. Similarly, in the case of a calculator 120 other than the conventional AI-PCC method, the guide surface feature derivation unit 121 reflects the set of guide surface features g(C^in) in the input coordinate set and input feature set {C^in, F^in}, and inputs the input coordinate set and input feature set {C^in, [F^in, g(C^in)]} to the calculation unit 122. This calculation unit 122 may have any configuration, such as a 3D-CNN, a multilayer perceptron, or other neural network.

For example, when points 141 exist in a three-dimensional space 140 as shown in FIG. 6, in the conventional method, the information of each point 141 is encoded. In contrast, in the case of method 1, a guide surface 142 is provided. This guide surface 142 is a virtual surface in a three-dimensional space set in the present technology, which is used to convert the information of each point 141. This guide surface 142 may be any surface, and may be a flat surface or a curved surface. In addition, this guide surface 142 may represent the three-dimensional shape (outer shape) of the same 3D object as the point cloud with a predetermined resolution, or may be a surface unrelated to the outer shape of the 3D object. In the case of method 1, the information of each point 141 is converted into information (guide surface feature amount) based on this guide surface 142 and encoded. Therefore, in 2D encoding, the amount of information to be encoded can be reduced, as in the case of encoding a prediction residual. Therefore, it is possible to suppress a decrease in encoding efficiency.

For example, the first information processing device is provided with a guide surface feature derivation unit that uses a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set; a coordinate encoding unit that encodes the derived second coordinate set; and a feature encoding unit that encodes the derived second feature set.

In addition, in a first information processing method executed by a first information processing device, a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, are used to derive guide surface features, which are features based on the positional relationship between the guide surface and the points, and a predetermined calculation is performed using the first coordinate set and the first feature set reflecting the guide surface features to derive a second coordinate set and a second feature set, the derived second coordinate set is encoded, and the derived second feature set is encoded.

By doing this, as described above, it is possible to reduce the amount of information to be encoded, and to prevent a decrease in encoding efficiency.

The scale of the second coordinate set and the second feature set may be smaller than the scale of the first coordinate set and the first feature set. In the first information processing device, the calculation unit may derive the second coordinate set and the second feature set by recursively repeating a predetermined calculation multiple times. The calculation unit may downscale the input coordinate set and feature set in all or a part of the multiple predetermined calculations. The guide surface feature may be derived for each scale. For example, the guide surface feature derivation unit may derive guide surface feature for some or all scales of the coordinate set and feature set.

The guide surface may also be encoded. The bit stream may also be decoded to generate (restore) the guide surface. The guide surface may also be generated. For example, the first information processing device may further include a guide surface generation unit that generates a guide surface using a point cloud, a guide surface encoding unit that encodes the guide surface and generates encoded data for the guide surface, and a guide surface decoding unit that decodes the encoded data for the guide surface and generates a guide surface.

The first information processing device may further include a score encoding unit that encodes the score.
The score encoding unit may also encode the score for each scale.

Furthermore, in the first information processing device, the calculation unit may be configured with a CNN (Convolution Neural Network). In that case, the first information processing device may further include a sparse tensor construction unit that constructs a sparse tensor from the point cloud. In that case, the CNN may input the constructed sparse tensor.

The second information processing device also includes a coordinate decoding unit that decodes encoded data to generate a first coordinate set, which is a set of coordinates of points forming a point cloud; a feature decoding unit that decodes encoded data to generate a first feature set, which is a set of features of the coordinates; a guide surface feature derivation unit that uses the generated first coordinate set, the generated first feature set, and a guide surface that is a reference surface formed in three-dimensional space to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; and a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features to derive a second coordinate set and a second feature set, so that the code amount of the encoded data of the first coordinate set and the first feature set is smaller than the code amount of the encoded data of the second coordinate set and the second feature set.

In addition, in a second information processing method executed by a second information processing device, encoded data is decoded to generate a first coordinate set which is a set of coordinates of points forming a point cloud, encoded data is decoded to generate a first feature set which is a set of coordinate features, the generated first coordinate set, the generated first feature set, and a guide surface which is a reference surface formed in three-dimensional space are used to derive guide surface features which are features based on the positional relationship between the guide surface and the points, a predetermined calculation is performed using the first coordinate set and the first feature set reflecting the guide surface features, and a new coordinate set and feature set are derived, and the amount of code of the encoded data of the first coordinate set and the first feature set is made smaller than the amount of code of the encoded data of the second coordinate set and the second feature set.

By doing this, as described above, it is possible to reduce the amount of information to be encoded, and to prevent a decrease in encoding efficiency.

The scale of the second coordinate set and the second feature set may be larger than the scale of the first coordinate set and the first feature set. In the second information processing device, the calculation unit may derive the second coordinate set and the second feature set by recursively repeating a predetermined calculation multiple times. The calculation unit may upscale the input coordinate set and feature set in all or a part of the multiple predetermined calculations. The guide surface feature may be derived for each scale. For example, the guide surface feature derivation unit may derive guide surface feature for some or all scales of the coordinate set and feature set.

The second information processing device may also decode and generate (restore) the guide surface. For example, the second information processing device may further include a guide surface decoding unit that decodes the encoded data of the guide surface and generates the guide surface.

The second information processing device may also include an occupancy state classification unit that predicts an occupancy probability value for each coordinate and extracts some coordinates and features with high occupancy probability values. The second information processing device may also include a score decoding unit that decodes to generate (restore) a score. The score decoding unit may also decode to generate (restore) a score for each scale.

In addition, in the second information processing device, the calculation unit may be configured using a Convolution Neural Network (CNN).

<Encoding device>
The first information processing device may be any device. For example, it may be an encoding device that encodes 3D data (e.g., a point cloud). FIG. 7 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 300 shown in FIG. 7 is a device that encodes the geometry of a point cloud (3D data). The encoding device 300 encodes the geometry by applying the above-mentioned method 1.

Note that FIG. 7 shows the main processing units, data flows, etc., and is not necessarily all that is shown in FIG. 7. In other words, in the encoding device 300, there may be processing units that are not shown as blocks in FIG. 7, and there may be processing and data flows that are not shown as arrows, etc. in FIG. 7.

As shown in FIG. 7, the encoding device 300 has a guide surface generation unit 311, a guide surface encoding unit 312, a guide surface decoding unit 313, a sparse tensor construction unit 314, a guide surface feature derivation unit 315, a CNN 316 (CNN_E1), a guide surface feature derivation unit 317, a CNN 318 (CNN_E2), a CNN 319 (CNN_E3), a guide surface feature derivation unit 320, a CNN 321 (CNN_E4), a CNN 322 (CNN_E5), a CNN 323 (CNN_E6), a coordinate encoding unit 324, a feature encoding unit 325, and a score encoding unit 326.

The guide surface generation unit 311 generates a guide surface from the input point cloud C_X. The guide surface is a reference surface formed in three-dimensional space, and is used for encoding the point cloud. For example, the guide surface generation unit 311 may generate a mesh from the point cloud and use the mesh as the guide surface. Any method may be used to generate a mesh from this point cloud, and an existing method such as Poisson surface reconstruction described in Non-Patent Document 2 may be applied. However, this method requires normal data, so it is necessary to calculate the normals of the point cloud. The guide surface generation unit 311 supplies data indicating the generated guide surface to the guide surface encoding unit 312.

The guide surface encoding unit 312 encodes the supplied guide surface to generate encoded data of the guide surface (also referred to as a guide surface bit stream). Any encoding method may be used. For example, if the guide surface is composed of a mesh, the guide surface encoding unit 312 may encode the guide surface by applying an encoding method for meshes. For example, an existing method such as reducing the number of triangles by mesh decimation and then performing Draco encoding may be applied. The guide surface encoding unit 312 may output the generated guide surface bit stream to the outside of the encoding device 300. The guide surface encoding unit 312 may also supply the generated guide surface bit stream to the guide surface decoding unit 313.

The guide surface decoding unit 313 decodes the supplied guide surface bit stream and generates (restores) the guide surface. This decoding method may be any method that corresponds to the encoding method applied by the guide surface encoding unit 312. For example, if the guide surface is configured by a mesh, the guide surface decoding unit 313 may apply a decoding method for meshes to decode the guide surface bit stream. This decoding method may be an existing method. For example, if the guide surface is a mesh, Draco decoding may be applied. Note that the encoding and decoding of this guide surface may be a lossless method or a lossy method. In other words, the guide surface generated (restored) by the guide surface decoding unit 313 (also referred to as a decoded guide surface) may not completely match the guide surface before encoding (for example, the guide surface generated by the guide surface generating unit 311). The guide surface decoding unit 313 may supply the generated decoded guide surface to the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320.

The sparse tensor construction unit 314 constructs an input sparse tensor X = {C_X, F_X} from the input point group C_X. A sparse tensor is a data structure for representing a set of feature vectors arranged in space. Each feature vector is associated with a coordinate in space. Specifically, a sparse tensor in three-dimensional space is represented by a pair {C, F} of coordinate set C = {(x_i, y_i, z_i)}_i and feature set F = {f_i}_i. Here, feature f_i is arranged at coordinates (x_i, y_i, z_i). Note that a dummy feature consisting of a series of values "1" is applied to the input feature set F_X. The scale of the input sparse tensor X = {C_X, F_X} constructed by the sparse tensor construction unit 314 is 1x (x1). The sparse tensor construction unit 314 may supply the constructed input sparse tensor X = {C_X, F_X} to the guide surface feature derivation unit 315.

The guide surface feature derivation unit 315 performs processing related to the derivation of guide surface features. For example, the guide surface feature derivation unit 315 receives the sparse tensor {C^in, F^in} to be processed, and derives the guide surface feature g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed. The guide surface feature derivation unit 315 then concatenates the obtained guide surface feature set g(C^in) = {g(c_i)}_i to the feature set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result. Here, [F^in, g(C^in)] is a feature set obtained by concatenating the guide surface feature g(c_i) calculated from the coordinate values corresponding to each feature f_i ∈ F^in. Define [F^in, g(C^in)] = {[f_i, g(c_i)]}_i. One method of calculating the guide surface feature quantity g(c_i) is, for example, to search for the nearest point p_i on the guide surface from the coordinate value c_i, and set g(c_i) = p_i - c_i ∈ R^3. In other words, in this case, the dimension of the feature quantity set [F^in, g(C^in)] resulting from the calculation is three more than the dimension of the feature quantity set F^in to be processed.

The guide surface feature derivation unit 315 may obtain the input sparse tensor X = {C_X, F_X} supplied from the sparse tensor construction unit 314 as the sparse tensor to be processed. The guide surface feature derivation unit 315 may also obtain the decoded guide surface supplied from the guide surface decoding unit 313. The guide surface feature derivation unit 315 may then use the decoded guide surface to perform the above-mentioned processing on the input sparse tensor to generate a sparse tensor as the calculation result. The guide surface feature derivation unit 315 may also supply the generated sparse tensor as the calculation result to the CNN 316 (CNN_E1).

CNN316 (CNN_E1) is a sparse 3D-CNN. Sparse 3D-CNN is a 3D-CNN with a sparse 3D convolutional layer, which inputs the sparse tensor {C^in, F^in} to be processed and outputs the sparse tensor {C^out, F^out} as the result of the calculation. 3D-CNN is a neural network mainly composed of a 3D convolutional layer and a nonlinear transformation layer. The 3D convolutional layer performs a kernel convolution operation on a set of features arranged in a three-dimensional space, and outputs the set of features as the result of the calculation. Roughly speaking, the 3D convolutional layer performs an operation to aggregate nearby feature sets in three-dimensional space by weighting each feature with the corresponding kernel value. This realizes feature transformation that takes into account nearby features. 3D-CNN realizes more nonlinear and complex feature transformation by stacking 3D convolutional layers and nonlinear transformation layers. A sparse 3D convolutional layer is a 3D convolutional layer whose input and output are sparse tensors. A sparse 3D convolutional layer has the advantage of being computationally more efficient (because empty coordinates with no corresponding features are ignored) than a normal 3D convolutional layer (also called a dense 3D convolutional layer for comparison). For example, the calculation of a sparse 3D convolutional layer can be expressed as the following equation (1).

　・・・（１）

...(1)

Note that C^in is the input coordinate set (also called the coordinate set of the processing target), and C^out is the output coordinate set (also called the coordinate set of the calculation result). f_u^in is the input feature vector at coordinate u (also called the feature vector of the processing target), and f_u^out is the output feature vector at coordinate u (also called the feature vector of the calculation result). N^3(u, C^in) = {i|u + i ∈ C^in, i ∈ N^3} defines the 3D convolution kernel and is the set of offset values i to the neighboring coordinates in C^in centered on coordinate u. W_i represents the kernel value at offset value i. Also, the coordinate set of the processing target C^in and the coordinate set of the calculation result C^out are not always the same. For example, C^out is different from C^in when downscaling or upscaling.

CNN316 (CNN_E1) may obtain the sparse tensor of the calculation result of the guide surface feature derivation unit 315 as the sparse tensor to be processed. In other words, since the dimension of the feature increases by the amount of the guide surface feature, the input/output dimension of each sparse 3D-CNN is changed accordingly. This also applies to the following CNNs (CNN318, CNN319, CNN321, CNN322, and CNN323). CNN316 (CNN_E1) is composed of two sparse 3D convolution layers and two ReLU (Rectified Linear Unit) layers, which are a type of nonlinear transformation layer, and may perform 1/2 downscaling on the sparse tensor to be processed. CNN316 (CNN_E1) may supply the sparse tensor of the calculation result obtained to the guide surface feature derivation unit 317. The CNN 316 (CNN_E1) may supply N_X = |C_X|, which corresponds to the number of points in the coordinate set C_X, to the point encoding unit 326.

The guide surface feature derivation unit 317 is a processing unit similar to the guide surface feature derivation unit 315, and performs processing related to the derivation of guide surface features. In other words, as described above, the guide surface feature derivation unit 317 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature set g(C^in) = {g(c_i)}_i to the feature set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

The guide surface feature derivation unit 317 may obtain a sparse tensor resulting from the calculation of the CNN 316 (CNN_E1) as a sparse tensor to be processed. The guide surface feature derivation unit 317 may also obtain a decoded guide surface supplied from the guide surface decoding unit 313. The guide surface feature derivation unit 317 may then use the decoded guide surface to perform the above-mentioned processing on the obtained sparse tensor to generate a sparse tensor resulting from the calculation. The guide surface feature derivation unit 317 may supply the sparse tensor resulting from the calculation that has been generated to the CNN 318 (CNN_E2).

CNN318 (CNN_E2) is a sparse 3D-CNN, similar to CNN316 (CNN_E1). CNN318 (CNN_E2) may obtain the sparse tensor resulting from the calculation of the guide surface feature derivation unit 317 as the sparse tensor to be processed. CNN318 (CNN_E2) may be configured with a three-layer IRN (Inception-Residual Network). The IRN is also a sparse 3D-CNN, and is configured with a sparse 3D convolutional layer and a nonlinear transformation layer. CNN318 (CNN_E2) may perform calculations using the three-layer IRN on the sparse tensor to be processed. CNN318 (CNN_E2) may supply the sparse tensor X' = [C_X', F_X'] resulting from the calculation to CNN319 (CNN_E3). Note that the scale of this sparse tensor X' = [C_X', F_X'] is 1/2 (x1/2).

The CNN319 (CNN_E3) is a sparse 3D-CNN, similar to the CNN316 (CNN_E1). The CNN319 (CNN_E3) may obtain the sparse tensor resulting from the calculation of the CNN318 (CNN_E2) as the sparse tensor to be processed. The CNN319 (CNN_E3) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform 1/2 downscaling on the sparse tensor to be processed. The CNN319 (CNN_E3) may supply the sparse tensor resulting from the calculation to the guide surface feature derivation unit 320. The CNN319 (CNN_E3) may supply N_X' = |C_X'|, which corresponds to the points of the coordinate set C_X' obtained by downscaling the coordinate set C_X by 1/2, to the point encoding unit 326.

The guide surface feature derivation unit 320 is a processing unit similar to the guide surface feature derivation unit 315, and performs processing related to the derivation of guide surface features. In other words, as described above, the guide surface feature derivation unit 320 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature set g(C^in) = {g(c_i)}_i to the feature set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

The guide surface feature derivation unit 320 may obtain the sparse tensor resulting from the calculation of CNN 319 (CNN_E3) as the sparse tensor to be processed. The guide surface feature derivation unit 320 may also obtain the decoded guide surface supplied from the guide surface decoding unit 313. The guide surface feature derivation unit 320 may then use the decoded guide surface to perform the above-described processing on the sparse tensor to be processed, to generate a sparse tensor resulting from the calculation. The guide surface feature derivation unit 320 may supply the sparse tensor resulting from the calculation that has been generated to CNN 321 (CNN_E4).

The CNN321 (CNN_E4) is a sparse 3D-CNN, similar to the CNN318 (CNN_E2). The CNN321 (CNN_E4) may obtain the sparse tensor resulting from the calculation of the guide surface feature derivation unit 320 as the sparse tensor to be processed. The CNN321 (CNN_E4) may be configured with three layers of IRNs, and may perform calculations using the three layers of IRNs on the sparse tensor to be processed. The CNN321 (CNN_E4) may supply the sparse tensor X'' = [C_X'', F_X''] resulting from the calculation to the CNN322 (CNN_E5). The scale of this sparse tensor X'' = [C_X'', F_X''] is 1/4 times (x1/4).

CNN322 (CNN_E5) is a sparse 3D-CNN, similar to CNN316 (CNN_E1). CNN322 (CNN_E5) may obtain the sparse tensor resulting from the computation of CNN321 (CNN_E4) as the sparse tensor to be processed. CNN322 (CNN_E5) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform 1/2 downscaling on the sparse tensor to be processed. CNN322 (CNN_E5) may supply the sparse tensor resulting from the computation obtained to CNN323 (CNN_E6). CNN322 (CNN_E5) may supply N_X'' = |C_X''|, which corresponds to the points of the coordinate set C_X'' obtained by downscaling the coordinate set C_X by 1/4, to the point encoding unit 326.

CNN323 (CNN_E6) is a sparse 3D-CNN, similar to CNN318 (CNN_E2). CNN323 (CNN_E6) may obtain the sparse tensor resulting from the computation of CNN322 (CNN_E5) as the sparse tensor to be processed. CNN323 (CNN_E6) may be configured with three layers of IRN and one layer of sparse 3D convolutional layer, and may perform computations on the sparse tensor to be processed using the three layers of IRN and one layer of sparse 3D convolutional layer. CNN323 (CNN_E6) may supply the sparse tensor Y = {C_Y, F_Y} resulting from the computation to the coordinate encoding unit 324 and the feature encoding unit 325. CNN323 (CNN_E6) may also supply the coordinate set C_Y resulting from the computation to the coordinate encoding unit 324, and the feature set F_Y resulting from the computation to the feature encoding unit 325. Note that the scale of this sparse tensor Y = {C_Y, F_Y} is 1/8 times (x1/8).

The coordinate encoding unit 324 is a G-PCC (Geometry-based Point Cloud Compression) encoder, and performs processing related to encoding of coordinates (coordinate set). The coordinate encoding unit 324 may acquire a sparse tensor Y = {C_Y, F_Y} (or its coordinate set C_Y) supplied from the CNN 323 (CNN_E6). The coordinate encoding unit 324 may encode the coordinate set C_Y and generate encoded data (also called a coordinate bit stream) of the coordinate set. In this case, the coordinate encoding unit 324 may encode the coordinate set C_Y using a lossless method. For example, the coordinate encoding unit 324 may encode the coordinate set C_Y using an octree. The coordinate encoding unit 324 may output the generated coordinate bit stream to the outside of the encoding device 300.

The feature encoding unit 325 performs processing related to encoding of features (feature set). The feature encoding unit 325 may acquire a sparse tensor Y = {C_Y, F_Y} (or feature set F_Y) supplied from the CNN 323 (CNN_E6). The feature encoding unit 325 may quantize and entropy encode the feature set F_Y to generate encoded data of the feature set (also referred to as a feature bit stream). In other words, the feature encoding unit 325 encodes the feature set F_Y using a lossy method. The feature encoding unit 325 may output the generated feature bit stream to the outside of the encoding device 300.

The score encoding unit 326 performs processing related to encoding the scores of the coordinate sets of each scale. The score encoding unit 326 may obtain N_X = |C_X| corresponding to the scores of the coordinate set C_X supplied from CNN 316 (CNN_E1). The score encoding unit 326 may obtain N_X' = |C_X'| corresponding to the scores of the coordinate set C_X' obtained by downscaling the coordinate set C_X by 1/2 supplied from CNN 319 (CNN_E3). The score encoding unit 326 may obtain N_X'' = |C_X''| corresponding to the scores of the coordinate set C_X'' obtained by downscaling the coordinate set C_X by 1/4 supplied from CNN 322 (CNN_E5). The score encoding unit 326 may encode these N_X, N_X', and N_X'' to generate encoded data of the scores (also referred to as a score bit stream). The score encoding unit 326 may output the generated score bitstream outside the encoding device 300.

That is, in the encoding device 300, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, and the guide surface feature derivation unit 320 derive guide surface feature amounts, which are features based on the positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space. Also, CNN316, CNN318, CNN319, CNN321, CNN322, and CNN323 perform a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface feature amounts (using features including the guide surface feature amounts) to derive a second coordinate set and a second feature set. Therefore, CNN316, CNN318, CNN319, CNN321, CNN322, and CNN323 can also be said to be calculation units. Furthermore, the coordinate encoding unit 324 encodes the derived second coordinate set. Furthermore, the feature encoding unit 325 encodes the derived second feature set. The amount of code for the encoded data of the second coordinate set and the second feature set is smaller than the amount of code for the encoded data of the first coordinate set and the first feature set.

By having such a configuration, the encoding device 300 can reduce the amount of information to be encoded, and suppress a decrease in encoding efficiency.

Note that the CNN and guide surface feature derivation unit may have any configuration and are not limited to the example in FIG. 7. For example, there may be any number of CNNs and guide surface feature derivation units. Furthermore, the number of downscalings and the conversion rate for each downscaling may be any number. Furthermore, while FIG. 7 illustrates an example in which the calculation unit is 3D-CNN, the calculation unit may have any configuration, such as 3D-CNN, a multilayer perceptron, or other neural networks, as in the calculation unit 122 in FIG. 5.

Furthermore, the guide surface feature amount derivation unit only needs to be provided at least at the beginning of the CNN sequence. In addition, it may be provided anywhere in the CNN sequence. For example, as described above, the guide surface feature amount may be derived for each scale. For example, as shown in FIG. 7, the guide surface feature amount derivation unit 315 may derive the guide surface feature amount at 1/1 scale, the guide surface feature amount derivation unit 317 may derive the guide surface feature amount at 1/2 scale, and the guide surface feature amount derivation unit 320 may derive the guide surface feature amount at 1/4 scale. Although omitted in the example of FIG. 7, the guide surface feature amount may also be derived for 1/8 scale. In other words, the scale of the second coordinate set and the second feature amount set may be smaller than the scale of the first coordinate set and the first feature amount set. Furthermore, the calculation unit may derive the second coordinate set and the second feature amount set by recursively repeating a predetermined calculation multiple times. The calculation unit may also downscale the input coordinate set and feature set in all or part of the multiple predetermined calculations. The guide surface feature derivation unit may then derive guide surface feature values for some or all of the scales of the coordinate set and feature set.

The guide surface may also be encoded. The bit stream may also be decoded to generate (restore) the guide surface. The guide surface may also be generated. For example, as shown in FIG. 7, the guide surface generation unit 311 may generate a guide surface using a point cloud, the guide surface encoding unit 312 may encode the guide surface to generate encoded data for the guide surface, and the guide surface decoding unit 313 may decode the encoded data for the guide surface to generate the guide surface.

<Encoding process flow>
The encoding device 300 performs an encoding process to encode the geometry as described above. An example of the flow of the encoding process will be described with reference to the flowcharts of FIGS.

When the encoding process is started, the guide surface generation unit 311 and the sparse tensor construction unit 314 acquire the input point group C_X, which is the point cloud to be input to the encoding device 300, in step S301 of FIG. 8.

In step S302, the guide surface generation unit 311 generates a guide surface from the input point group C_X.

In step S303, the guide surface encoding unit 312 encodes the generated guide surface to generate a guide surface bit stream. The guide surface encoding unit 312 outputs the guide surface bit stream to the outside of the encoding device 300.

In step S304, the guide surface decoding unit 313 decodes the guide surface bitstream and generates (restores) the decoded guide surface.

In step S305, the sparse tensor construction unit 314 generates an input sparse tensor X={C_X, F_X} (scaled 1x) from the input point group C_X.

In step S306, the guide surface feature derivation unit 315 sets {C^in, F^in} ← (C_X, F_X). In other words, the guide surface feature derivation unit 315 sets the input sparse tensor X = {C_X, F_X} as the sparse tensor (C^in, F^in) to be processed.

In step S307, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, or the guide surface feature derivation unit 320 determines whether or not to input the guide surface feature to the next 3D-CNN. If it is determined that the guide surface feature is to be input to the next 3D-CNN, the process proceeds to step S308.

In step S308, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, or the guide surface feature derivation unit 320 executes a guide surface feature set derivation process, and derives a guide surface feature set g(C^in) using the sparse tensor to be processed and the decoded guide surface.

In step S309, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, or the guide surface feature derivation unit 320 sets F^in ← [F^in, g(C^in)]. In other words, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, or the guide surface feature derivation unit 320 sets the concatenation of the feature set to be processed and the guide surface feature set g(C^in) as the feature set to be processed by the next 3D-CNN.

When the processing of step S309 is completed, the processing proceeds to FIG. 9. Also, in step S307, if it is determined that the guide surface feature amount is not to be input to the next 3D-CNN, the processing proceeds to FIG. 9. In other words, if there is no guide surface feature amount derivation unit immediately before the 3D-CNN, the processing of steps S308 and S309 is omitted. For example, in the case where the encoding device 300 has the configuration example shown in FIG. 7, when the next 3D-CNN is CNN316 (CNN_E1), CNN318 (CNN_E2), or CNN321 (CNN_E4), the processing of steps S308 and S309 is executed, and when the next 3D-CNN is CNN319 (CNN_E3), CNN322 (CNN_E5), or CNN323 (CNN_E6), the processing of steps S308 and S309 is omitted.

In step S321 of FIG. 9, CNN316 (CNN_E1), CNN318 (CNN_E2), CNN319 (CNN_E3), CNN321 (CNN_E4), CNN322 (CNN_E5), or CNN323 (CNN_E6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

In step S322, CNN323 (CNN_E6) determines whether or not all 3D-CNNs have been processed. If it is determined that all 3D-CNNs have not been processed (i.e., there are unprocessed 3D-CNNs (at least CNN323 (CNN_E6) has not been processed)), the process proceeds to step S323.

In step S323, CNN316 (CNN_E1), CNN318 (CNN_E2), CNN319 (CNN_E3), CNN321 (CNN_E4), or CNN322 (CNN_E5) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN316 (CNN_E1), CNN318 (CNN_E2), CNN319 (CNN_E3), CNN321 (CNN_E4), or CNN322 (CNN_E5) sets the sparse tensor {C^out, F^out} resulting from the calculation it derived as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or 3D-CNN).

When the processing of step S323 is completed, the process returns to step S307 in FIG. 8, and the subsequent processing is executed. That is, the processing of each of steps S307 to S309 in FIG. 8 and the processing of each of steps S321 to S323 in FIG. 9 are executed for each 3D-CNN (including the guide surface feature derivation unit if one exists immediately before the 3D-CNN).

If it is determined in step S322 of FIG. 9 that processing has been performed for all 3D-CNNs (i.e., processing has been performed for CNN323 (CNN_E6)), processing proceeds to step S324.

In step S324, CNN323 (CNN_E6) sets {C_Y, F_Y} ← {C^out, F^out}. In other words, the last CNN323 (CNN_E6) sets the sparse tensor {C^out, F^out}, which is the result of the calculation it derived, as the output sparse tensor {C_Y, F_Y}.

In step S325, the coordinate encoding unit 324 encodes the coordinate set C_Y of the output sparse tensor {C_Y, F_Y} to generate a coordinate bitstream. It also outputs the coordinate bitstream to the outside of the encoding device 300.

In step S326, the feature encoding unit 325 encodes the feature set F_Y of the output sparse tensor {C_Y, F_Y} to generate a feature bitstream. The feature bitstream is then output to the outside of the encoding device 300.

In step S327, the scores for each scale are encoded to generate a score bit stream. The score bit stream is then output to the outside of the encoding device 300.

When step S327 is completed, the encoding process ends.

<Flow of guide surface feature set derivation process>
Next, an example of the flow of the guide surface feature amount set derivation process executed in step S308 in FIG. 8 will be described with reference to the flowchart in FIG.

When the guide surface feature amount set derivation process is started, in step S331, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 extracts coordinates c_i ∈ C^in. In other words, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 selects one coordinate c_i from the coordinate set C^in to be processed, and makes it the processing target.

In step S332, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 searches for the nearest point p_i on the decoded guide surface from the selected coordinate c_i.

In step S333, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 returns g(c_i) = p_i - c_i. In other words, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 derives the guide surface feature amount g(c_i) based on the positional relationship between the guide surface and the point (in this example, p_i - c_i).

In step S334, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, or the guide surface feature derivation unit 320 determines whether or not processing has been performed for all coordinates c_i ∈ C^in. If it is determined that an unprocessed coordinate c_i exists, the process returns to step S331, and the subsequent processes are executed. In other words, the processes of steps S331 to S334 are executed for each coordinate value c_i in the coordinate set C^in to be processed.

If it is determined in step S334 that processing has been performed for all coordinate values c_i, the guide surface feature set derivation process ends and processing returns to FIG. 8.

By performing each process in this manner, the encoding device 300 can reduce the amount of information to be encoded and suppress a decrease in encoding efficiency.

<Decoding device>
Fig. 11 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 350 shown in Fig. 11 is a device that decodes a bit stream in which the geometry of a point cloud (3D data) is encoded. For example, the decoding device 350 decodes bit streams (guide surface bit stream, coordinate bit stream, feature bit stream, and point number bit stream) generated by encoding (the geometry of) the point cloud by the encoding device 300, and generates (restores) (the geometry of) the point cloud.

Note that FIG. 11 shows the main processing units, data flows, etc., and is not necessarily all that is shown in FIG. 11. In other words, in the decoding device 350, there may be processing units that are not shown as blocks in FIG. 11, and there may be processing and data flows that are not shown as arrows, etc. in FIG. 11.

As shown in FIG. 11, the decoding device 350 has a score decoding unit 361, a coordinate decoding unit 362, a feature decoding unit 363, a guide surface decoding unit 364, a CNN 365 (CNN_D1), a guide surface feature derivation unit 366, a CNN 367 (CNN_D2), an occupancy state classification unit 368, a CNN 369 (CNN_D3), a guide surface feature derivation unit 370, a CNN 371 (CNN_D4), an occupancy state classification unit 372, a CNN 373 (CNN_D5), a guide surface feature derivation unit 374, a CNN 375 (CNN_D6), and an occupancy state classification unit 376.

The score decoding unit 361 executes processing related to the decoding of the score bit stream. For example, the score decoding unit 361 may acquire the score bit stream input to the decoding device 350. The score decoding unit 361 may decode the acquired score bit stream and generate (restore) information indicating the scores of the coordinate sets of each scale. For example, the score decoding unit 361 may decode the score bit stream and generate (restore) N_X = |C_X| corresponding to the scores of the coordinate set C_X before encoding, which are encoded by the score encoding unit 326 (FIG. 7), N_X' = |C_X'| corresponding to the scores of the coordinate set C_X' obtained by downscaling the coordinate set C_X before encoding by 1/2, and N_X'' = |C_X''| corresponding to the scores of the coordinate set C_X'' obtained by downscaling the coordinate set C_X before encoding by 1/4. The score decoding unit 361 may supply the generated N_X to the occupancy state classification unit 376. The score decoding unit 361 may supply the generated N_X' to the occupancy state classification unit 372. The score decoding unit 361 may supply the generated N_X'' to the occupancy state classification unit 368.

The coordinate decoding unit 362 is a G-PCC decoder and executes processing related to decoding of the coordinate bit stream. For example, the coordinate decoding unit 362 may acquire the coordinate bit stream input to the decoding device 350. The coordinate decoding unit 362 may decode the acquired coordinate bit stream and generate (restore) the coordinate set C_Y. In this case, the coordinate decoding unit 362 may decode the coordinate bit stream using a lossless method. For example, the coordinate decoding unit 362 may decode the coordinate bit stream using an octree. The coordinate decoding unit 362 may supply the generated coordinate set C_Y to the CNN 365 (CNN_D1).

The feature decoding unit 363 executes processing related to the decoding of the feature bit stream. For example, the feature decoding unit 363 may acquire the feature bit stream input to the decoding device 350. The feature decoding unit 363 may entropy decode the acquired feature bit stream to generate (restore) a feature set F_Y. In this case, the feature decoding unit 363 decodes the feature bit stream using a lossy method. Therefore, the feature set F_Y generated by the feature decoding unit 363 does not have to completely match the feature set F_Y before being encoded by the feature encoding unit 325 (Figure 7). The feature decoding unit 363 may supply the generated feature set F_Y to the CNN 365 (CNN_D1).

The guide surface decoding unit 364 is a processing unit similar to the guide surface decoding unit 313 (FIG. 7), and executes processing related to the decoding of the guide surface bit stream. For example, the guide surface decoding unit 364 may acquire the guide surface bit stream input to the decoding device 350. The guide surface decoding unit 364 may decode the acquired guide surface bit stream and generate a decoded guide surface. This decoding method may be any method that corresponds to the encoding method applied by the guide surface encoding unit 312 (FIG. 7). For example, when the guide surface is composed of a mesh, the guide surface decoding unit 364 may decode the guide surface bit stream by applying a decoding method for meshes. This decoding method may be an existing method. For example, when the guide surface is a mesh, Draco decoding may be applied. The encoding and decoding of the guide surface may be a lossless method or a lossy method. In other words, the guide surface (also referred to as the decoded guide surface) generated (restored) by the guide surface decoding unit 364 does not have to completely match the guide surface before encoding (for example, the guide surface generated by the guide surface generating unit 311). The guide surface decoding unit 364 may supply the generated decoded guide surface to the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374.

CNN365 (CNN_D1) is a sparse 3D-CNN, and is mainly composed of sparse 3D convolution layers and nonlinear transformation layers. It acquires a sparse tensor as a processing target, performs a predetermined operation on the sparse tensor to be processed, and outputs a sparse tensor as the result of the operation. For example, CNN365 (CNN_D1) may acquire a coordinate set C_Y supplied from the coordinate decoding unit 362 and a feature set F_Y supplied from the feature decoding unit 363 as a sparse tensor Y = {C_Y, F_Y} to be processed. The scale of this sparse tensor Y = {C_Y, F_Y} is 1/8 times (x1/8) that before encoding. CNN365 (CNN_D1) is composed of two sparse 3D convolution layers and two ReLU layers, and may perform upscaling by 2 times on the sparse tensor to be processed. The CNN 365 (CNN_D1) may supply the sparse tensor of the obtained calculation result to the guide surface feature derivation unit 366.

The guide surface feature derivation unit 366 is a processing unit similar to the guide surface feature derivation units of the encoding device 300 such as the guide surface feature derivation unit 315, and performs processing related to the derivation of guide surface features. The guide surface feature derivation unit 366 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature set g(C^in) = {g(c_i)}_i to the feature set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result. As with the encoding device 300, one method for calculating the guide surface feature quantity g(c_i) is to search for the nearest point p_i on the guide surface from the coordinate value c_i, and make g(c_i) = p_i - c_i ∈ R^3. In other words, in this case, the dimension of the feature quantity set [F^in, g(C^in)] resulting from the calculation is three more than the dimension of the feature quantity set F^in to be processed.

The guide surface feature derivation unit 366 may obtain the sparse tensor resulting from the calculation of CNN 365 (CNN_D1) as the sparse tensor to be processed. The guide surface feature derivation unit 366 may also obtain the decoded guide surface supplied from the guide surface decoding unit 364. The guide surface feature derivation unit 366 may then use the decoded guide surface to perform the above-mentioned processing on the sparse tensor to be processed, to generate a sparse tensor resulting from the calculation. The guide surface feature derivation unit 366 may also supply the sparse tensor resulting from the calculation that has been generated to CNN 367 (CNN_D2).

CNN367 (CNN_D2), like CNN365 (CNN_D1), is a sparse 3D-CNN, and is mainly composed of a sparse 3D convolution layer and a nonlinear transformation layer. In other words, CNN367 (CNN_D2) acquires a sparse tensor as a processing target, performs a predetermined operation on the sparse tensor to be processed, and outputs a sparse tensor as the operation result. For example, CNN367 (CNN_D2) may acquire a sparse tensor as the operation result of the guide surface feature derivation unit 366 as the sparse tensor to be processed. However, since the sparse tensor to be processed has an increased dimension by the guide surface feature, the input/output dimensions of CNN367 (CNN_D2) are also changed accordingly. CNN367 (CNN_D2) is composed of a three-layer IRN, and may perform an operation using the three-layer IRN on the sparse tensor to be processed. The CNN 367 (CNN_D2) may supply the sparse tensor {Up(C_Y), F_X''} obtained as the result of the calculation to the occupancy state classification unit 368. Note that Up() represents a process of upsampling by a factor of two in the x, y, and z directions. For example, the score of Up(C_Y) is eight times the score of C_Y.

The occupancy state classification unit 368 is composed of one sparse 3D convolutional layer. This sparse 3D convolutional layer inputs the sparse tensor to be processed and outputs the sparse tensor that is the result of the calculation. The feature values of the sparse tensor that is the result of the calculation of this sparse 3D convolutional layer represent the occupancy probability value of the coordinates. The occupancy state classification unit 368 extracts a predetermined number of coordinates (top k coordinates) from those with the highest occupancy probability value, and deletes the other coordinates. In other words, the occupancy state classification unit 368 outputs a coordinate set in which the coordinate set of the input sparse tensor has been deleted except for the top k coordinates, and a feature set corresponding to the coordinate set (i.e., the feature set of the input sparse tensor in which the feature values corresponding to the deleted coordinates have been deleted). Note that the scale value corresponding to the sparse tensor to be processed, among N_X, N_X', and N_X'', is applied to this k. The occupancy state classification unit 368 also deletes the feature values corresponding to the deleted coordinates in F^out.

The occupancy state classification unit 368 may obtain the sparse tensor {Up(C_Y), F_X''} resulting from the calculation of the CNN 367 (CNN_D2) as the sparse tensor to be processed. The occupancy state classification unit 368 may obtain N_X'' = |C_X''| corresponding to the scores of the coordinate set C_X'' obtained by downscaling the coordinate set C_X before encoding by 1/4, which is supplied from the score decoding unit 361. The occupancy state classification unit 368 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X'' coordinate values of the occupancy probability value, delete the other coordinate values, and generate the sparse tensor X'' = {C_X'', F_X''} resulting from the calculation. The occupancy state classification unit 368 may supply the sparse tensor X'' = {C_X'', F_X''} resulting from the calculation to the CNN 369 (CNN_D3).

CNN369 (CNN_D3), like CNN365 (CNN_D1), is a sparse 3D-CNN, and is mainly composed of sparse 3D convolutional layers and nonlinear transformation layers. In other words, CNN369 (CNN_D3) acquires a sparse tensor as a processing target, performs a predetermined operation on the sparse tensor to be processed, and outputs a sparse tensor as the operation result. For example, CNN369 (CNN_D3) may acquire the sparse tensor X'' = {C_X'', F_X''}, which is the operation result of the occupancy state classification unit 368, as the sparse tensor to be processed. The scale of this sparse tensor X'' = {C_X'', F_X''} is 1/4 times (x1/4) that before encoding. CNN369 (CNN_D3) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform upscaling by 2 times on the sparse tensor to be processed. The CNN 369 (CNN_D3) may supply the sparse tensor of the obtained calculation result to the guide surface feature derivation unit 370.

The guide surface feature derivation unit 370 is a processing unit similar to the guide surface feature derivation unit 366, and performs processing related to the derivation of guide surface features. As with the guide surface feature derivation unit 366, the guide surface feature derivation unit 370 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature set g(C^in) = {g(c_i)}_i to the feature set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

The guide surface feature derivation unit 370 may obtain the sparse tensor resulting from the calculation of CNN 369 (CNN_D3) as the sparse tensor to be processed. The guide surface feature derivation unit 370 may also obtain the decoded guide surface supplied from the guide surface decoding unit 364. The guide surface feature derivation unit 370 may then use the decoded guide surface to perform the above-mentioned processing on the sparse tensor to be processed, to generate a sparse tensor resulting from the calculation. The guide surface feature derivation unit 370 may also supply the sparse tensor resulting from the calculation that has been generated to CNN 371 (CNN_D4).

CNN371 (CNN_D4) is a sparse 3D-CNN, similar to CNN367 (CNN_D2), and is mainly composed of a sparse 3D convolution layer and a nonlinear transformation layer. In other words, CNN371 (CNN_D4) acquires a sparse tensor as a processing target, performs a predetermined operation on the sparse tensor to be processed, and outputs a sparse tensor as the operation result. For example, CNN371 (CNN_D4) may acquire a sparse tensor as the operation result of the guide surface feature derivation unit 370 as the sparse tensor to be processed. However, since the sparse tensor to be processed has an increased dimension by the guide surface feature, the input/output dimensions of CNN371 (CNN_D4) are also changed accordingly. CNN371 (CNN_D4) is composed of a three-layer IRN, and may perform an operation using the three-layer IRN on the sparse tensor to be processed. The CNN 371 (CNN_D4) may supply the sparse tensor {Up(C_X''), F_X'} obtained as the calculation result to the occupancy state classification unit 372.

The occupancy state classification unit 372 is a processing unit similar to the occupancy state classification unit 368, and is composed of one sparse 3D convolutional layer. It predicts the occupancy probability value of the sparse tensor to be processed, extracts a predetermined number of coordinates (the top k coordinates) with the highest occupancy probability values, and deletes the remaining coordinates. The occupancy state classification unit 372 also deletes the features in F^out that correspond to the deleted coordinates.

For example, the occupancy state classification unit 372 may obtain the sparse tensor {Up(C_X'', F_X'} resulting from the calculation of the CNN 371 (CNN_D4) as the sparse tensor to be processed. The occupancy state classification unit 372 may obtain N_X' = |C_X'|, which corresponds to the scores of the coordinate set C_X' obtained by downscaling the coordinate set C_X before encoding by 1/2, supplied from the score decoding unit 361. The occupancy state classification unit 372 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X' coordinate values of the occupancy probability value, delete the other coordinate values, and generate the sparse tensor X' = {C_X', F_X'} resulting from the calculation. The occupancy state classification unit 372 may supply the sparse tensor X' = {C_X', F_X'} resulting from the calculation to the CNN 373 (CNN_D5).

CNN373 (CNN_D5), like CNN365 (CNN_D1), is a sparse 3D-CNN, and is mainly composed of sparse 3D convolutional layers and nonlinear transformation layers. In other words, CNN373 (CNN_D5) acquires a sparse tensor as the processing target, performs a predetermined operation on the sparse tensor to be processed, and outputs a sparse tensor as the operation result. For example, CNN373 (CNN_D5) may acquire the sparse tensor X' = {C_X', F_X'}, which is the operation result of the occupancy state classification unit 372, as the sparse tensor to be processed. The scale of this sparse tensor X' = {C_X', F_X'} is 1/2 times (x1/2) that before encoding. CNN373 (CNN_D5) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform upscaling by 2 times on the sparse tensor to be processed. The CNN 373 (CNN_D5) may supply the sparse tensor of the obtained calculation result to the guide surface feature derivation unit 374.

The guide surface feature derivation unit 374 is a processing unit similar to the guide surface feature derivation unit 366, and performs processing related to the derivation of guide surface features. As with the guide surface feature derivation unit 366, the guide surface feature derivation unit 374 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature set g(C^in) = {g(c_i)}_i to the feature set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

The guide surface feature derivation unit 374 may obtain the sparse tensor resulting from the calculation of CNN 373 (CNN_D5) as the sparse tensor to be processed. The guide surface feature derivation unit 374 may also obtain the decoded guide surface supplied from the guide surface decoding unit 364. The guide surface feature derivation unit 374 may then use the decoded guide surface to perform the above-mentioned processing on the sparse tensor to be processed, to generate a sparse tensor resulting from the calculation. The guide surface feature derivation unit 374 may also supply the sparse tensor resulting from the calculation that has been generated to CNN 375 (CNN_D6).

CNN375 (CNN_D6) is a sparse 3D-CNN, similar to CNN367 (CNN_D2), and is mainly composed of a sparse 3D convolution layer and a nonlinear transformation layer. In other words, CNN375 (CNN_D6) acquires a sparse tensor as a processing target, performs a predetermined operation on the sparse tensor to be processed, and outputs a sparse tensor as the operation result. For example, CNN375 (CNN_D6) may acquire a sparse tensor as the operation result of the guide surface feature derivation unit 374 as the sparse tensor to be processed. However, since the sparse tensor to be processed has an increased dimension by the guide surface feature, the input/output dimensions of CNN375 (CNN_D6) are also changed accordingly. CNN375 (CNN_D6) is composed of a three-layer IRN, and may perform an operation using the three-layer IRN on the sparse tensor to be processed. The CNN 375 (CNN_D6) may supply the sparse tensor {Up(C_X'), F_X} obtained as the calculation result to the occupancy state classification unit 376.

The occupancy state classification unit 376 is a processing unit similar to the occupancy state classification unit 368, and is composed of one sparse 3D convolutional layer. It predicts the occupancy probability value of the sparse tensor to be processed, extracts a predetermined number of coordinates (the top k coordinates) with the highest occupancy probability values, and deletes the remaining coordinates. The occupancy state classification unit 376 also deletes the features in F^out that correspond to the deleted coordinates.

For example, the occupancy state classification unit 376 may obtain the sparse tensor {Up(C_X'), F_X} resulting from the computation of the CNN 375 (CNN_D6) as the sparse tensor to be processed. The occupancy state classification unit 376 may obtain N_X = |C_X| corresponding to the points of the coordinate set C_X before encoding, supplied from the score decoding unit 361. The occupancy state classification unit 376 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X coordinate values of the occupancy probability value, and delete the other coordinate values to generate the sparse tensor X = {C_X, F_X} resulting from the computation. The scale of this sparse tensor X = {C_X, F_X} is 1x (x1) the scale before encoding. In other words, the occupancy state classification unit 376 may generate (restore) a point cloud obtained by decoding the bit stream group (score bit stream, guide surface bit stream, coordinate bit stream, and feature bit stream). This point cloud is also referred to as the decoded point group C^X. The occupancy state classification unit 376 may output the decoded point group C_X resulting from the calculation to outside the decoding device 350.

That is, in the decoding device 350, the coordinate decoding unit 362 decodes the encoded data to generate a first coordinate set, which is a set of coordinates of points forming a point cloud. The feature decoding unit 363 decodes the encoded data to generate a first feature set, which is a set of features of the coordinates. The guide surface feature derivation unit 366, the guide surface feature derivation unit 370, and the guide surface feature derivation unit 374 use the first coordinate set, the first feature set, and the guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points. The CNNs 365, 367, 369, CNN 371, CNN 373, and CNN 375 perform a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set. Therefore, CNN365, CNN367, CNN369, CNN371, CNN373, and CNN375 can also be considered calculation units.

By having such a configuration, the decoding device 350 can suppress a decrease in encoding efficiency.

Note that the configurations of the CNN, guide surface feature derivation unit, occupancy state classification unit, etc. may be any configuration and are not limited to the example of FIG. 11. For example, there may be any number of CNNs, guide surface feature derivation units, and occupancy state classification units. Furthermore, the number of downscalings and the conversion rate for each downscaling may be any number. Furthermore, while FIG. 11 illustrates an example in which the calculation unit is 3D-CNN, the calculation unit may have any configuration, such as 3D-CNN, a multilayer perceptron, or other neural networks, as in the calculation unit 122 of FIG. 5.

The guide surface feature amount derivation unit may be provided anywhere in the CNN sequence. For example, as described above, the guide surface feature amount may be derived for each scale. For example, as shown in FIG. 11, the guide surface feature amount derivation unit 366 may derive the guide surface feature amount at 1/4 scale, the guide surface feature amount derivation unit 370 may derive the guide surface feature amount at 1/2 scale, and the guide surface feature amount derivation unit 374 may derive the guide surface feature amount at 1/1 scale. Although omitted in the example of FIG. 11, the guide surface feature amount may also be derived for 1/8 scale. That is, the scale of the second coordinate set and the second feature amount set may be larger than the scale of the first coordinate set and the first feature amount set. The calculation unit may derive the second coordinate set and the second feature amount set by recursively repeating a predetermined calculation multiple times. The calculation unit may upscale the input coordinate set and feature amount set in all or part of the multiple predetermined calculations. The guide surface feature derivation unit may then derive guide surface feature values for some or all of the scales of the coordinate set and feature set.

The guide surface may also be generated (restored) by decoding. For example, as shown in FIG. 11, the guide surface decoding unit 364 may decode the encoded data of the guide surface and generate the guide surface.

<Decryption process flow>
The decoding device 350 executes the decoding process to decode the bit stream and generate (restore) the geometry as described above. An example of the flow of this decoding process will be described with reference to the flowcharts of Figs. 12 and 13.

When the decoding process starts, the score decoding unit 361 decodes the score bit stream in step S351 of FIG. 12, and generates scores for each scale (N_X, N_X', N_X'').

In step S352, the coordinate decoding unit 362 decodes the coordinate bit stream and generates a coordinate set C_Y.

In step S353, the feature decoding unit 363 decodes the feature bit stream and generates a feature set F_Y.

In step S354, the guide surface decoding unit 364 decodes the guide surface bitstream and generates a decoded guide surface.

In step S355, CNN365 (CNN_D1) sets {C^in, F^in} ← {C_Y, F_Y}. In other words, CNN365 (CNN_D1) sets the sparse tensor Y = {C_Y, F_Y} as the sparse tensor {C^in, F^in} to be processed.

In step S356, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) determines whether or not to input guide surface features to its own 3D-CNN. If it is determined that the guide surface features are to be input, the process proceeds to step S357. In other words, if the CNN to be processed is CNN367 (CNN_D2), CNN371 (CNN_D4), or CNN375 (CNN_D6), the process proceeds to step S357.

In step S357, the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, or the guide surface feature amount derivation unit 374 executes a guide surface feature amount set derivation process to derive the guide surface feature amount set g(C^in). This guide surface feature amount set derivation process is executed in the same manner as described with reference to the flowchart in FIG. 10.

In step S358, the guide surface feature derivation unit 366, the guide surface feature derivation unit 370, or the guide surface feature derivation unit 374 sets F^in ← [F^in, g(C^in)]. In other words, the guide surface feature derivation unit 366, the guide surface feature derivation unit 370, or the guide surface feature derivation unit 374 sets the concatenation of the feature set to be processed and the guide surface feature set g(C^in) as the feature set to be processed by the next 3D-CNN.

When the processing of step S358 ends, the process proceeds to FIG. 13. Also, if it is determined in step S356 of FIG. 12 that guide surface features are not to be input to its own 3D-CNN, the processing of steps S357 and S358 is omitted, and the process proceeds to FIG. 13. In other words, if the CNN to be processed is CNN365 (CNN_D1), CNN369 (CNN_D3), or CNN373 (CNN_D5), the processing of steps S357 and S358 is omitted, and the process proceeds to FIG. 13.

In step S371 of FIG. 13, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

In step S372, the occupancy state classification unit 368, the occupancy state classification unit 372, or the occupancy state classification unit 376 determines whether or not to classify the occupancy state. If it is determined that the occupancy state is to be classified, the process proceeds to step S373. In other words, if the process of step S371 is executed by CNN 367 (CNN_D2), CNN 371 (CNN_D4), or CNN 375 (CNN_D6), the process proceeds to step S373.

In step S373, the occupancy state classification unit 368, the occupancy state classification unit 372, or the occupancy state classification unit 376 predicts the occupancy probability of each coordinate in the coordinate set C^out, extracts the top k coordinates with the highest occupancy probability, and deletes the remaining coordinates. At this time, the scale value corresponding to the sparse tensor to be processed, among N_X, N_X', or N_X'', is applied to this k. In addition, the occupancy state classification unit 368, the occupancy state classification unit 372, or the occupancy state classification unit 376 also deletes the features corresponding to the deleted coordinates in F^out.

When the process of step S373 ends, the process proceeds to step S374. Also, if it is determined in step S372 that occupancy state classification is not to be performed, the process of step S373 is omitted and the process proceeds to step S374. In other words, if the process of step S371 is performed by CNN365 (CNN_D1), CNN369 (CNN_D3), or CNN373 (CNN_D5), the process of step S373 is omitted and the process proceeds to step S374.

In step S374, CNN375 (CNN_D6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., there are unprocessed 3D-CNNs (at least CNN375 (CNN_D6) is unprocessed)), processing proceeds to step S375.

In step S375, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) sets the sparse tensor {C^out, F^out} of the calculation result derived by itself as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or occupancy state classification unit). When the processing of step S375 is completed, the process returns to step S356 in FIG. 12, and subsequent processing is executed. That is, the processes of steps S356 to S358 in FIG. 12 and steps S371 to S375 in FIG. 13 are executed for each 3D-CNN (which may include a guide surface feature derivation unit and an occupancy state classification unit).

If it is determined in step S374 of FIG. 13 that processing has been performed for all 3D-CNNs, processing proceeds to step S376.

In step S376, the occupancy state classification unit 376 sets C_X ← C^out. In other words, the occupancy state classification unit 376 sets the coordinate set C^out of the calculation result as the decoding result (decoded point group C_X).

In step S377, the occupancy state classification unit 376 outputs the decoded point group C_X to the outside of the decoding device 350.

When step S377 is completed, the decoding process ends.

By performing each process in this manner, the decoding device 350 can suppress a decrease in encoding efficiency.

<Method 1-1>
In addition, when the above-mentioned method 1 is applied, as shown in the second row from the top of the table in FIG. 4, the guide surface feature amount may be a feature amount based on the nearest point on the guide surface from the point (method 1-1).

For example, in the first information processing device or the second information processing device, the guide surface feature amount may be a feature amount based on the nearest point on the guide surface with respect to the point.

<Method 1-1-1>
In addition, when the above-mentioned method 1-1 is applied, the guide surface feature amount may be a vector from a point to the nearest point, as shown in the third row from the top of the table in FIG. 4 (method 1-1-1). For example, as shown in FIG. 14, for each coordinate c_i = (x_i, y_i, z_i) of the coordinate set C^in, the nearest point p_i on the guide surface 142 is derived. The vector p_i - c_i ∈ R^3 is set as the guide surface feature amount. That is, in this case, g(c_i) = p_i - c_i. By doing so, the decoder can roughly utilize information such as how far the coordinate is from the guide surface.

<Method 1-1-2>
Furthermore, when the above-mentioned method 1-1 is applied, as shown in the fourth row from the top of the table in FIG. 4, the guide surface feature amount may be a feature amount relating to the distance between a point and a nearest neighboring point (method 1-1-2).

For example, the guide surface feature may be a signed distance. A signed distance is a scalar value obtained by multiplying the distance from a point to a boundary surface by +1 if the point is included within the boundary, and by -1 if not. In other words, if the distance from coordinate c_i to the nearest point p_i on the guide surface is expressed as ||p_i - c_i||, then g(c_i) is defined as the following equation (2).

　・・・（２）

... (2)

However, signed distance can only be defined for faces that have an inside and an outside. For example, if the guide surface is a mesh, the mesh must be watertight (in simple terms, a mesh without holes). If the mesh is not watertight, the unsigned distance g(c_i) = ||p_i - c_i|| may be applied instead of the signed distance. The unsigned distance simply indicates the distance only, and can be calculated even for meshes that are not watertight.

Also, a truncated signed distance, which limits the range of absolute values in the signed distance, may be applied as a guide surface feature. In other words, a certain maximum absolute value τ is determined, and if the absolute value of the signed distance is greater than τ, it is truncated to -τ or τ.

<Method 1-1-3>
Furthermore, in the case where the above-mentioned method 1-1 is applied, as shown in the fifth row from the top of the table in FIG. 4, the guide surface feature amount may be the feature amount of the nearest point (method 1-1-3).

In other words, some feature value h(p_i) at the nearest point p_i on the guide surface may be applied as the guide surface feature value g(c_i). In this case, g(c_i) = h(p_i) is defined. This feature value h(p_i) may be anything. For example, it may be the normal vector at the nearest point p_i, or it may be the curvature of the guide surface at the nearest point p_i (a numerical value that indicates the degree of curvature of the surface, and Gaussian curvature or mean curvature can be specifically calculated).

<Method 1-2>
The guide surface, which is a reference surface formed in a three-dimensional space, may be configured with any 3D data. For example, when the above-mentioned method 1 is applied, the guide surface may be a mesh (method 1-2), as shown in the sixth row from the top of the table in FIG. 4.

For example, in the first information processing device and the second information processing device, the guide surface may be a mesh.

Generally, a mesh contains less information than a point cloud. By encoding guide surface features that use the mesh as a guide surface (reference surface), the amount of information to be encoded can be reduced compared to encoding the point cloud as is. This makes it possible to prevent a decrease in encoding efficiency.

<Method 1-2-1>
Also, when the guide surface is a mesh, the guide surface may be output as 3D data of the decoding result. For example, the restoration of the point cloud may be omitted, and the decoded guide surface may be output as the decoding result. In this way, the increase in the load of the decoding process can be suppressed. Also, the decoded guide surface may be fitted with the restored point cloud. That is, when the above-mentioned method 1-2 is applied, the guide surface may be fitted to the point cloud as shown in the seventh row from the top of the table in FIG. 4 (method 1-2-1).

For example, three-dimensional space 380A and three-dimensional space 380B in FIG. 15 are two-dimensional representations of three-dimensional space. In these three-dimensional spaces 380A and 380B, grey squares 381 indicate points of the restored point cloud. Although only one square is marked in FIG. 15, all grey squares indicate points of the point cloud. In contrast, assume that guide surface 382A is restored like three-dimensional space 380A. If the guide surface is a mesh, as in this example, there may be cases where the mesh surfaces and vertices are misaligned with respect to the point cloud. In other words, if such a guide surface is used as the decoded result, the resolution will be lower than in the case of a point cloud, and there is a risk that the quality of the decoded result will be reduced.

Therefore, like guide surface 382B in three-dimensional space 380B, fitting may be performed on the guide surface using points of the point cloud (so that the surfaces and vertices of the guide surface are brought closer to the points of the point cloud). By doing so, it is possible to prevent a reduction in the definition of the guide surface, and therefore a reduction in the quality of the decoded results.

For example, the second information processing device may further include a fitting unit that fits the guide surface to the point cloud.

<Decoding device>
When such method 1-2-1 is applied, the configuration of the encoding device and the flow of the decoding process are similar to those described with reference to FIGS.

FIG. 16 is a block diagram showing an example of the main configuration of a decoding device when this method 1-2-1 is applied. The decoding device 400 shown in FIG. 16 is a device that decodes a bit stream in which the geometry of a point cloud (3D data) is encoded, similar to the decoding device 350. For example, the decoding device 400 decodes the bit streams (guide surface bit stream, coordinate bit stream, feature bit stream, and point number bit stream) generated by encoding the point cloud (geometry) by the encoding device 300. However, the decoding device 400 outputs a decoded guide surface (mesh) as the decoding result.

Note that FIG. 16 shows the main processing units, data flows, etc., and is not necessarily all that is shown in FIG. 16. In other words, in the decoding device 400, there may be processing units that are not shown as blocks in FIG. 16, and there may be processing and data flows that are not shown as arrows, etc. in FIG. 16.

As shown in FIG. 16, the decoding device 400 has a decoding unit 411 and a fitting unit 412.

The decoding unit 411 has the same configuration as the decoding device 350 (FIG. 11) and performs the same processing. That is, the explanation given above with reference to FIGS. 11 to 13 can be applied to the decoding unit 411. The decoding unit 411 decodes the point bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream, and supplies the decoded point group C_X to the fitting unit 412. The decoding unit 411 also supplies the generated decoded guide surface (in this case, the guide surface is a mesh, so it is also called a decoded guide mesh) to the fitting unit 412.

The fitting unit 412 obtains the decoded point group C_X and the decoded guide mesh. The fitting unit 412 fits the decoded guide mesh to the decoded point group C_X.

Any method may be used for this fitting. For example, an existing method for fitting a mesh to a point cloud may be applied. For example, the fitting unit 412 may perform mesh subdivision, which subdivides polygons, on the guide mesh, and mesh smoothing, which is a smoothing process. The fitting unit 412 may then perform fitting using Template Deformation for Point Cloud Fitting, as described in Non-Patent Document 3, on the guide mesh whose surface shape has been smoothed by increasing the number of triangles. This method is a process for fitting a template mesh to a point cloud. In this case, the template mesh corresponds to the guide mesh, and the point cloud corresponds to the decoded point cloud.

However, as an initial condition, a small number of corresponding point pairs (information indicating which vertices of the template mesh correspond to which points in the point cloud) are required. Here, the nearest point in the point cloud is searched for from each vertex of the template mesh, and the top few pairs with the shortest distance to the nearest point can be used as the initial corresponding point pairs. Note that since no local rotation is performed here, no rotation is input, and fitting can be performed with a rotation angle of 0. To do this, for example, all rotation matrices in the algorithm can be set to unit matrices.

The fitting unit 412 outputs the guide mesh after fitting to the outside of the decoding device 400 as the decoding result.

By having such a configuration, the decoding device 400 can output the guide mesh as the decoding result while suppressing any reduction in the quality of the decoding result.

<Decryption process flow>
An example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart of FIG.

When the decoding process is started, in step S401, the decoding unit 411 executes the decoding process described with reference to Figures 8 and 9, and decodes the bit stream to generate a decoded point group and a guide mesh.

In step S402, the fitting unit 412 fits the guide mesh to the decoded point cloud. In step S403, the fitting unit 412 outputs the mesh resulting from the fitting to the outside of the decoding device 400 as a decoding result. When the processing of step S403 ends, the decoding processing ends.

By performing each process in this manner, the decoding device 400 can output the guide mesh as the decoding result while suppressing any reduction in the quality of the decoding result.

<Method 1-3>
Furthermore, when the above-mentioned method 1 is applied, the guide surface may be a triangle soup, as shown in the eighth row from the top of the table in FIG. 4 (method 1-3).

For example, in the first information processing device and the second information processing device, the guide surface may be a triangle soup.

In this case, the guide surface is stored as a type of mesh called a triangle soup, and is encoded using Trisoup coding. Trisoup coding is one option of G-PCC for encoding dense point clouds. The encoder estimates a set of triangles (=Triangle soup) that approximates the input point cloud as shown in Figure 18, and encodes the Triangle soup. The Triangle soup is the low-resolution Octree and the triangular data in each Octree node. The low-resolution Octree is encoded using Octree coding, and the intersections (also called vertices) between the triangles in each node and each edge are encoded. The decoder decodes the Triangle soup, samples the points from the surface, and outputs them.

<Encoding device>
An example of the main configuration of the encoding device 300 when applying this method 1-3 is shown in Fig. 19. As shown in Fig. 19, in this case, the encoding device 300 has a Trisoup encoding unit 431 instead of the guide surface generating unit 311 and the guide surface encoding unit 312 in the example of Fig. 7. Also, the encoding device 300 has a Trisoup decoding unit 432 instead of the guide surface decoding unit 313 in the example of Fig. 7.

The Trisoup encoding unit 431 performs Trisoup encoding on the input point group C_X, estimates the Triangle soup internally, and generates encoded data (guide surface bit stream). The Trisoup encoding unit 431 supplies the generated guide surface bit stream to the Trisoup decoding unit 432. The Trisoup encoding unit 431 also outputs the guide surface bit stream to the outside of the encoding device 300. In other words, the Trisoup encoding unit 431 has the functions of both the guide surface generation unit 311 and the guide surface encoding unit 312.

The Trisoup decoding unit 432 obtains the guide surface bit stream supplied from the Trisoup encoding unit 431. The Trisoup decoding unit 432 Trisoup decodes the guide surface bit stream, generates a Triangle soup internally, and generates a point cloud sampled from that surface. However, instead of the point cloud, the Trisoup decoding unit 432 uses the Triangle soup generated internally during the decoding as the decoded guide surface, and supplies this to the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, and the guide surface feature derivation unit 320.

By having such a configuration, the encoding device 300 can encode the guide surface as a triangle soup. In other words, in this case too, the encoding device 300 can reduce the amount of information to be encoded, and suppress a decrease in encoding efficiency.

As described above, the Trisoup encoding unit 431 encodes the low-resolution Octree. The coordinate encoding unit 324 performs Octree encoding and encodes the Octree corresponding to the coordinate set C_Y. These two Octrees are not necessarily identical, but if the two Octrees match, the encoding device 300 may encode and output one of the Octrees. In this case, the decoding device 350 can reuse one of the Octrees obtained by decoding as the other Octree. In this way, it is possible to suppress an increase in the amount of code and further suppress a decrease in encoding efficiency.

Also, when one octree matches an octree obtained by downscaling the other octree, the encoding device 300 may encode and output only the high-resolution octree. In that case, the decoding device 350 may downscale the high-resolution octree obtained by decoding to restore the low-resolution octree. In this way, it is possible to suppress an increase in the amount of code and further suppress a decrease in encoding efficiency.

<Encoding process flow>
An example of the flow of the encoding process executed by the encoding device 300 when Methods 1-3 are applied will be described with reference to the flowcharts of FIGS.

When the encoding process starts, the Trisoup encoding unit 431 and the sparse tensor construction unit 314 acquire the input point cloud C_X, which is the point cloud to be input to the encoding device 300, in step S431 of FIG. 20.

In step S432, the Trisoup encoding unit 431 performs Trisoup encoding on the input point group C_X to generate a guide plane bit stream. The Trisoup encoding unit 431 outputs the guide plane bit stream to the outside of the encoding device 300.

In step S433, the Triangle soup decoding unit 432 decodes the guide surface bit stream and generates a Triangle soup as the decoded guide surface.

The processes of steps S434 to S438 in FIG. 20 and steps S441 to S447 in FIG. 21 are executed in the same manner as the processes of steps S305 to S309 in FIG. 8 and steps S321 to S327 in FIG. 9. Then, when the process of step S447 in FIG. 21 ends, the encoding process ends.

By performing each process in this manner, the encoding device 300 can reduce the amount of information to be encoded and suppress a decrease in encoding efficiency.

<About the Decryption Side>
In addition, when applying this method 1-3, the decoding device 350 may include a Trisoup decoding unit instead of the guide surface decoding unit 364 in Fig. 11. This Trisoup decoding unit is a processing unit similar to the Trisoup decoding unit 432 of the encoding device 300, and performs similar processing. In other words, this Trisoup decoding unit acquires the guide surface bit stream supplied to the decoding device 350, Trisoup-decodes the guide surface bit stream, generates a Triangle soup inside, sets it as a decoded guide surface, and supplies it to the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374. The other configurations are the same as those in Fig. 11.

In this case, the flow of the decoding process executed by the decoding device 350 is basically the same as that described with reference to the flowcharts in Figures 12 and 13. However, in step S364, the Triangle soup decoding unit performs Triangle decoding on the guide surface bit stream, and the obtained Triangle soup is used as the decoded guide surface.

<Method 1-4>
Furthermore, when the above-mentioned method 1 is applied, the guide surface may be an implicit function expressing a three-dimensional shape, as shown in the ninth row from the top of the table in FIG. 4 (method 1-4).

For example, in the first information processing device and the second information processing device, the guide surface may be an implicit function.

The implicit function representation of a three-dimensional shape is a method of representing a three-dimensional shape by a set of coordinates where f(x, y, z) = 0, using a function f(x, y, z) for three-dimensional coordinate values (x, y, z). For example, a unit sphere can be represented as f(x, y, z) = x^2 + y^2 + z^2 - 1. When representing a general three-dimensional shape, for example, the signed distance from a point (x, y, z) to a guide surface may be represented as f(x, y, z). Similarly, the unsigned distance or the truncated signed distance may be represented as f(x, y, z). The occupancy probability value at a point (x, y, z) may also be represented as f(x, y, z). However, since the possible range of the occupancy probability value is [0, 1], in practice, an appropriately scaled value of the occupancy probability value may be represented as f(x, y, z).

<Method 1-4-1>
In addition, when the guide surface is an implicit function, the process of searching for the nearest point on the guide surface is generally not easy. Therefore, when the above-mentioned method 1-4 is applied, as shown in the 10th row from the top of the table in Figure 4, the guide surface feature amount may be the output of the implicit function at that coordinate or the gradient vector of that output (method 1-4-1).

For example, the implicit function value f(x_i, y_i, z_i) at the coordinates of the point c_i = (x_i, y_i, z_i) may be used as the guide surface feature g(c_i) (g(c_i) = f(x_i, y_i, z_i)). In addition, the value of the gradient vector ∇f at that coordinate value of the implicit function f or the Laplacian value Δf may also be applied. Roughly speaking, when the implicit function f is a signed distance function, the gradient vector ∇f on the guide surface coincides with the normal vector. For example, a vector concatenating all of the above features may be applied as the guide surface feature (g(c_i) = [f(x_i, y_i, z_i), ∇f(x_i, y_i, z_i), Δf(x_i, y_i, z_i)]).

<About the encoding device>
The configuration of the encoding device 300 when this method 1-4-1 is applied is the same as that described with reference to FIG. 7. However, the guide surface generating unit 311 estimates an implicit function as a guide surface from the input point group C_X. For example, the guide surface generating unit 311 generates a mesh from the input point group C_X, quantizes a three-dimensional space with an appropriate resolution (i.e., initializes a three-dimensional array), calculates the signed distance (or unsigned distance) from the center of each cell to the estimated mesh, and generates a three-dimensional array representing the signed distance. This three-dimensional array is supplied to the guide surface encoding unit 312 as a guide surface of the implicit function. This guide surface of the implicit function is also called a guide implicit function.

The guide surface encoding unit 312 encodes the guide implicit function to generate a guide surface bit stream. Any encoding method may be used. For example, the guide surface encoding unit 312 may apply an existing implicit function encoding method. For example, deep implicit volume compression as described in Non-Patent Document 4 may be applied. This method is capable of encoding a truncated signed distance.

The guide surface decoding unit 313 decodes the guide surface bit stream and generates (restores) a guide implicit function. Therefore, the obtained decoded guide surface is also called a decoded guide implicit function. Any decoding method may be applied as long as it corresponds to the encoding method of the guide surface encoding unit 312.

The decoded guide implicit function is supplied to the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320. The guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320 use the decoded guide implicit function to derive the guide surface feature amount. The guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320 may use the output of the implicit function at the coordinates of the point or the gradient vector of that output as this guide surface feature amount.

<About the encoding process>
The flow of the encoding process in this case is the same as that described with reference to the flowcharts of Figures 8 to 10. However, as described above in the description of the encoding device, in step S302, a guide implicit function is generated from the input point group. In step S303, the guide implicit function is encoded to generate a guide plane bit stream. In step S304, the guide plane bit stream is decoded to generate a decoded guide implicit function.

In addition, in step S308, the output of the implicit function at the coordinates of the point and the gradient vector of that output are derived as guide surface features.

<Decryption device>
The configuration of the decoding device 350 when applying this method 1-4-1 is the same as that described with reference to FIG. 11. However, the guide surface decoding unit 364 decodes the acquired guide surface bit stream and generates (restores) a guide implicit function. Any decoding method may be applied as long as it corresponds to the encoding method of the guide surface encoding unit 312. The guide surface decoding unit 364 supplies the generated decoded guide implicit function to the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374.

The guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374 use the decoded guide implicit function to derive the guide surface feature amount. The guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374 may use the output of the implicit function at the coordinates of the point or the gradient vector of that output as this guide surface feature amount.

<Decryption process>
The flow of the encoding process in this case is the same as that described with reference to the flowcharts of Fig. 12 and Fig. 13. However, as described above in the description of the decoding device, in step S354, the guide surface bit stream is decoded to generate a decoded guide implicit function. In addition, in step S357, the output of the implicit function at the coordinates of the point and the gradient vector of the output are derived as guide surface feature amounts.

<Method 1-5>
Furthermore, when the above-mentioned method 1 is applied, the parameter set to be applied may be selected as shown in the eleventh row from the top of the table in FIG. 4 (method 1-5).

For example, the first information processing device may further include a parameter set selection unit that selects a parameter set to be applied to the encoding of the point cloud. The first information processing device may further include a guide surface generation unit that generates a guide surface using the point cloud, and a guide surface encoding unit that applies the selected parameter set to encode the guide surface and generate encoded data for the guide surface.

In guide surface coding, there may be parameter sets that affect the coding efficiency and decoding error of the guide surface. For example, when using mesh decimation and Draco coding, examples of coding parameter sets include the target number of triangles in decimation (the smaller the number, the better the coding efficiency but the larger the error) and the quantization parameter (QP (Quantization Parameter)) in the Draco encoder (the smaller the number, the better the coding efficiency but the larger the error). The encoder may select and apply the parameter set that provides the highest coding efficiency for the point cloud from a candidate set of parameter sets (also called guide surface coding parameter sets) that are applied to coding of such guide surfaces.

By doing this, it is possible to suppress the reduction in coding efficiency due to the applied parameter set.

<Encoding device>
Fig. 22 is a block diagram showing a main configuration example of an encoding device when methods 1-5 are applied. The encoding device 500 shown in Fig. 22 is a device that encodes the geometry of a point cloud (3D data) in the same manner as the encoding device 300. The encoding device 500 encodes the geometry by applying the above-mentioned methods 1-5.

Note that FIG. 22 shows the main processing units, data flows, etc., and is not necessarily all that is shown in FIG. 22. In other words, in encoding device 500, there may be processing units that are not shown as blocks in FIG. 22, and there may be processing and data flows that are not shown as arrows, etc. in FIG. 22.

As shown in FIG. 22, the encoding device 500 has a parameter set supply unit 511, an encoding unit 512, a decoding unit 513, and a parameter set selection unit 514.

The parameter set supply unit 511 selects one parameter set from the candidate set of guide surface encoding parameter sets as the processing target, and supplies it to the encoding unit 512 and the parameter set selection unit 514.

The encoding unit 512 acquires the input point group C_X input to the encoding device 500. The encoding unit 512 also acquires the guide surface encoding parameter set supplied from the parameter set supply unit 511. The encoding unit 512 also applies the guide surface encoding parameter set to encode the input point group C_X and generate a bit stream. The encoding unit 512 may have a configuration similar to that of the encoding device 300 (FIG. 7), for example, and perform similar processing. That is, the encoding unit 512 encodes the input point group C_X and generates a point number bit stream, a coordinate bit stream, a feature amount bit stream, and a guide surface bit stream. At that time, the encoding unit 512 applies the guide surface encoding parameter set supplied from the parameter set supply unit 511 in the guide surface encoding unit 312. That is, the guide surface encoding unit 312 of the encoding unit 512 encodes the guide surface using the guide surface encoding parameter set supplied from the parameter set supply unit 511, and generates a guide surface bit stream. The encoding unit 512 supplies the bit streams generated in this manner (score bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream) to the decoding unit 513 and the parameter set selection unit 514.

The decoding unit 513 obtains the bit stream supplied from the encoding unit 512. The decoding unit 513 decodes the bit stream and generates (restores) the decoded point group C_X. The decoding unit 513 may have a configuration similar to that of the decoding device 350 (FIG. 11), for example, and may perform similar processing. That is, the decoding unit 513 decodes the point number bit stream, coordinate bit stream, feature amount bit stream, and guide surface bit stream, and generates the decoded point group C_X. The decoding unit 513 supplies the generated decoded point group C_X to the parameter set selection unit 514.

The parameter set selection unit 514 acquires a guide surface encoding parameter set supplied from the parameter set supply unit 511. The parameter set selection unit 514 acquires an input point group C_X input to the encoding device 500. The parameter set selection unit 514 acquires bit streams (e.g., a point number bit stream, a coordinate bit stream, a feature amount bit stream, and a guide surface bit stream) supplied from the encoding unit 512.
The parameter set selection unit 514 acquires the decoded point group C_X supplied from the decoding unit 513 .

The parameter set selection unit 514 selects a guide plane coding parameter set that maximizes the coding efficiency of the input point group C_X based on these. The parameter set selection unit 514 selects a bit stream corresponding to the selected guide plane coding parameter set and outputs it to the outside of the coding device 500.

In selecting this parameter set, the parameter set selection unit 514 may, for example, select the parameter set that minimizes the value of the RD cost function C = R + λ・D. Here, R indicates the total size of the bit stream. D indicates the error between the decoded geometry and the input geometry. The coefficient λ is a parameter that determines whether the bit size or the decoding error is to be emphasized. As an index of the error between point clouds, for example, the D1 metric, which is an index that measures the distance between point clouds, or the D2 metric, which also takes into account the normal direction, may be applied.

By selecting the guide surface coding parameter set to be applied so as to optimize it using the RD cost in this manner, the coding device 500 can suppress a decrease in coding efficiency.

<Encoding process flow>
An example of the flow of the encoding process executed by the encoding device 500 will be described with reference to the flowchart of FIG.

When the encoding process starts, in step S501, the parameter set supply unit 511 sets the coefficient λ of the RD cost function and a candidate set of guide surface encoding parameter sets.

In step S502, the parameter set supply unit 511 selects the i-th guide surface encoding parameter set from the set candidate set. In other words, the parameter set supply unit 511 selects one unprocessed guide surface encoding parameter set.

In step S503, the encoding unit 512 executes the encoding process described with reference to the flowcharts of, for example, FIG. 8 and FIG. 9. In this case, the encoding unit 512 applies the guide surface encoding parameters selected in step S502 to encode the guide surface and generate a guide surface bit stream. The encoding unit 512 then decodes the guide surface bit stream to generate a decoded guide surface, and encodes the point group to generate a point number bit stream, a coordinate bit stream, and a feature bit stream. When the guide surface encoding parameter set to be applied changes, the size of the guide surface bit stream and the decoded guide surface change. In other words, the overall bit stream size and the decoded point group C_X also change. Therefore, the encoding unit 512 derives not only the guide surface bit stream, but all bit streams.

In step S504, the parameter set selection unit 514 calculates the total size R_i of all bit streams (score bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream) generated in step S503.

In step S505, the decoding unit 513 executes the decoding process described with reference to the flowcharts in Figures 12 and 13, decodes the bit streams (point number bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream) generated in step S503, and generates a decoded point group C_X.

In step S506, the parameter set selection unit 514 derives the error D_i between the decoded geometry (the decoded point group C_X generated in step S505) and the input geometry (the input point group C_X).

In step S507, the parameter set selection unit 514 derives the value of the RD cost function C_i = R_i + λ·D_i using the coefficient λ set in step S501, the bitstream size R_i derived in step S504, and the error D_i derived in step S506.

In step S508, the parameter set selection unit 514 determines whether all parameter sets have been processed. If it is determined that an unprocessed parameter set exists, the process returns to step S502, and the subsequent processes are executed. In other words, the processes of steps S502 to S508 are executed for each parameter set, and the value C_i of the RD cost function is derived.

If it is determined in step S508 that processing has been performed for all parameter sets, processing proceeds to step S509.

In step S509, the parameter set selection unit 514 compares the RD cost function values C_i corresponding to each parameter set derived as described above, and selects the parameter set for which the RD cost function value C_i is the smallest.

In step S510, the parameter set selection unit 514 outputs a bit stream corresponding to the selected parameter set to the outside of the encoding device 500.

When the processing of step S510 is completed, the decoding process ends.

By performing each process as described above, the encoding device 500 can select a guide surface encoding parameter set that optimizes the RD cost and apply it to the encoding of the input point group C_X (for encoding the guide surface). Therefore, the encoding device 500 can suppress a decrease in encoding efficiency.

<About Decryption>
Note that even when Method 1-5 is applied, the decoding device 350 has the same configuration as in Fig. 11 and performs the same decoding process. That is, the explanation given with reference to Figs. 11 to 13 can be applied to this case as well.

<Method 1-6>
In addition, when the above-mentioned method 1 is applied, attributes may be transmitted (method 1-6) as shown in the bottom row of the table in Fig. 4. That is, although the transmission (encoding/decoding) of geometry has been described above, not only geometry but also attributes may be transmitted (encoding/decoding).

For example, in the first information processing device, the guide surface feature derivation unit may derive guide surface features using a coordinate set, a feature set including attribute information, and a guide surface including texture. Then, the calculation unit may perform calculations using the guide surface features to derive a new coordinate set and a feature set including attributes. Then, the coordinate encoding unit may encode the coordinate set to generate a coordinate bit stream. Also, the feature encoding unit may encode the feature set including attributes to generate a feature bit stream. Also, the guide surface encoding unit may encode the guide surface including texture to generate a guide surface bit stream. Also, the score encoding unit may encode information indicating the score of each scale to generate a score bit stream.

For example, in the second information processing device, the coordinate decoding unit may decode the coordinate bit stream to generate a coordinate set. The feature decoding unit may decode the feature bit stream to generate a feature set including attribute information. The guide surface decoding unit may decode the guide surface bit stream to generate a decoded guide surface including texture. The guide surface feature derivation unit may derive guide surface features using the coordinate set, the feature set including attributes, and the decoded guide surface including texture. The calculation unit may then perform calculations using the derived guide surface features to derive a new coordinate set and feature set including attributes.

Let us consider the case where attributes are attached to each point in the input point cloud. The attributes can be any kind of information. For example, they can include color information or reflected brightness. Such point cloud attributes can be transmitted (encoded and decoded) in the same framework as geometry.

<Encoding device>
Fig. 24 is a block diagram showing an example of the main configuration of an encoding device when methods 1-6 are applied. The encoding device 600 shown in Fig. 24 is a device that encodes a point cloud (3D data) in the same manner as the encoding device 300. The encoding device 600 performs encoding by applying the above-mentioned methods 1-6. That is, the encoding device 600 encodes attributes together with geometry.

Note that FIG. 24 shows the main processing units, data flows, etc., and is not necessarily all that is shown in FIG. 24. In other words, in the encoding device 600, there may be processing units that are not shown as blocks in FIG. 24, and there may be processing and data flows that are not shown as arrows, etc. in FIG. 24.

As shown in FIG. 24, the encoding device 600 has a guide surface generation unit 611, a guide surface encoding unit 612, a guide surface decoding unit 613, a sparse tensor construction unit 614, a guide surface feature derivation unit 615, a CNN 616 (CNN_E1), a guide surface feature derivation unit 617, a CNN 618 (CNN_E2), a CNN 619 (CNN_E3), a guide surface feature derivation unit 620, a CNN 621 (CNN_E4), a CNN 622 (CNN_E5), a CNN 623 (CNN_E6), a coordinate encoding unit 624, a feature encoding unit 625, and a score encoding unit 626.

The input point group input to the encoding device 600 is composed of geometry and attributes. In other words, each point has coordinate values (x, y, z) and attribute values (e.g., RGB values in the case of color information).

The guide surface generation unit 611 is a processing unit similar to the guide surface generation unit 311 (Figure 7) and performs similar processing. However, the guide surface generation unit 611 obtains an input point group composed of geometry and attributes, and generates a guide surface including the attributes. For example, if the guide surface is composed of a mesh, the attributes are formed as a texture. Therefore, in this case, the guide surface generation unit 611 generates a guide surface including a texture.

One way to store textures is to store the texture data as an image and add texture coordinate values (values that indicate where that vertex is located in the texture image) (also called UV coordinates) to each vertex of the mesh.

In addition to generating a guide surface (mesh) from the input point cloud, the guide surface generator 611 must also generate a texture from the input attributes. For each point in the input point cloud, the guide surface generator 611 searches for the nearest point on the guide mesh and copies the attributes of that point to the texel (one pixel in texture data is called a texel) that corresponds to that nearest point. However, after the above processing, there is a possibility that empty texels will exist. In that case, the guide surface generator 611 may use neighboring values to complement the empty texel value.

The guide surface encoding unit 612 is a processing unit similar to the guide surface encoding unit 312 (FIG. 7) and performs similar processing. However, the guide surface encoding unit 612 encodes the guide surface including the texture supplied from the guide surface generation unit 611 to generate a guide surface bit stream. That is, the guide surface encoding unit 612 needs to encode the texture data in addition to encoding the mesh data. That is, the guide surface encoding unit 612 not only encodes the geometry but also encodes the texture coordinates and texture image of each vertex. Any encoding method may be applied to encode the texture coordinates of each vertex. For example, Draco encoding may be applied (the Draco encoder can encode the texture coordinates in addition to the mesh). In addition, an existing image compression method (for example, JPEG (Joint Photographic Experts Group) or PNG (Portable Network Graphics)) may be applied to encode the texture image. Therefore, the guide surface bit stream in this case includes a bit stream that encodes the texture in addition to the bit stream that encodes the mesh.

The guide surface decoding unit 613 is a processing unit similar to the guide surface decoding unit 313 (FIG. 7) and performs similar processing. However, the guide surface decoding unit 613 decodes the guide surface bit stream and generates a decoded guide surface including texture. In other words, the guide surface decoding unit 613 decodes the bit stream to not only generate geometry, but also generate texture coordinates and texture images for each vertex. Any decoding method may be applied to the decoding when generating texture coordinates for each vertex. For example, Draco decoding may be applied (a Draco decoder can generate texture coordinates in addition to meshes). In addition, an existing image decoding method (for example, JPEG, PNG, etc.) may be applied to the decoding when generating texture images.

The sparse tensor construction unit 614 is a processing unit similar to the sparse tensor construction unit 314 (Figure 7) and performs similar processing. However, the sparse tensor construction unit 614 constructs an input sparse tensor X = {C_X, F_X} from an input point group consisting of geometry and attributes. In this case, in order to input attributes, the feature set F_X is a vector that lists the attributes of each coordinate. In other words, each feature f_i ∈ F_X is the attribute value of each point (e.g., RGB value). In addition, since the dimension of the feature vector increases, the input dimension of the CNN 616 (CNN_E1) must also be increased accordingly.

The guide surface feature amount derivation unit 615 is a processing unit similar to the guide surface feature amount derivation unit 315 (Figure 7) and performs similar processing. However, in order to use attribute information on the texture for encoding and decoding, the guide surface feature amount derivation unit 615 also includes features extracted from the texture in the guide surface feature amount. Specifically, the guide surface feature amount derivation unit 615 concatenates the texture values a_i (e.g., RGB values for color information) at the nearest point p_i on the guide mesh. In other words, the vectors are concatenated to obtain g(c_i) = [p_i - c_i, a_i].

CNN616 (CNN_E1) is a processing unit similar to CNN316 (CNN_E1) (Figure 7) and performs similar processing. However, in this case, in the sparse tensor processed by CNN616 (CNN_E1), F_X is changed to a vector that lists the attributes of each coordinate. In other words, each feature f_i ∈ F_X is the attribute value of each point (for example, RGB value). In addition, because the dimension of the feature vector increases, the input dimension of CNN616 (CNN_E1) must also be increased accordingly.

The guide surface feature amount derivation unit 617 is a processing unit similar to the guide surface feature amount derivation unit 317 (Figure 7) and performs similar processing. However, like the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617 also includes features extracted from the texture in the guide surface feature amount in order to use attribute information on the texture for encoding and decoding.

CNN618 (CNN_E2) is a processing unit similar to CNN318 (CNN_E2) (Figure 7) and performs similar processing. However, in this case, it is necessary to increase the input dimensions of CNN618 (CNN_E2), as in the case of CNN616 (CNN_E1).

CNN619 (CNN_E3) is a processing unit similar to CNN319 (CNN_E3) (Figure 7) and performs similar processing.

The guide surface feature amount derivation unit 620 is a processing unit similar to the guide surface feature amount derivation unit 320 (Figure 7) and performs similar processing. However, like the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 620 also includes features extracted from the texture in the guide surface feature amount in order to use attribute information on the texture for encoding and decoding.

CNN621 (CNN_E4) is a processing unit similar to CNN321 (CNN_E4) (Figure 7) and performs similar processing. However, in this case, it is necessary to increase the input dimensions of CNN621 (CNN_E4), as in the case of CNN616 (CNN_E1).

CNN622 (CNN_E5) is a processing unit similar to CNN322 (CNN_E5) (Figure 7) and performs similar processing.

CNN623 (CNN_E6) is a processing unit similar to CNN323 (CNN_E6) (Figure 7) and performs similar processing.

The coordinate encoding unit 624 is a processing unit similar to the coordinate encoding unit 324 (Figure 7) and performs similar processing. The feature encoding unit 625 is a processing unit similar to the feature encoding unit 325 (Figure 7) and performs similar processing. In this case, however, the feature set encoded by the feature encoding unit 625 includes not only geometry information but also attribute information. The score encoding unit 626 is a processing unit similar to the score encoding unit 326 (Figure 7) and performs similar processing.

With this configuration, the encoding device 600 can encode not only geometry but also attributes.

<Encoding process flow>
An example of the flow of the encoding process executed by the encoding device 600 will be described with reference to the flowcharts of FIGS.

In this case, when the encoding process is started, the guide surface generation unit 611 and the sparse tensor construction unit 614 acquire the input point group C_X, which is the point cloud input to the encoding device 600, and its attributes in step S601 of FIG. 25.

In step S602, the guide surface generation unit 611 generates a guide surface and texture data from the input point group C_X and attributes.

In step S603, the guide surface encoding unit 612 encodes the generated guide surface and texture data to generate a guide surface bit stream. The guide surface encoding unit 612 outputs the guide surface bit stream to the outside of the encoding device 600.

In step S604, the guide surface decoding unit 613 decodes the guide surface bitstream and generates (restores) a decoded guide surface and decoded texture data.

In step S605, the sparse tensor construction unit 614 generates an input sparse tensor X={C_X, F_X} (scaled 1x) from the input point group C_X and attributes.

In step S606, the guide surface feature derivation unit 615 sets {C^in, F^in} ← (C_X, F_X). In other words, the guide surface feature derivation unit 615 sets the input sparse tensor X = {C_X, F_X} as the sparse tensor (C^in, F^in) to be processed.

In step S607, the guide surface feature derivation unit 615, the guide surface feature derivation unit 617, or the guide surface feature derivation unit 620 determines whether or not to input the guide surface feature to the next 3D-CNN. If it is determined that the guide surface feature is to be input to the next 3D-CNN, the process proceeds to step S608.

In step S608, the guide surface feature derivation unit 615, the guide surface feature derivation unit 617, or the guide surface feature derivation unit 620 executes a guide surface feature set derivation process, and derives a guide surface feature set g(C^in) using the sparse tensor to be processed, the decoded guide surface, and the decoded texture data.

In step S609, the guide surface feature derivation unit 615, the guide surface feature derivation unit 617, or the guide surface feature derivation unit 620 sets F^in ← [F^in, g(C^in)]. In other words, the guide surface feature derivation unit 615, the guide surface feature derivation unit 617, or the guide surface feature derivation unit 620 sets the concatenation of the feature set to be processed and the guide surface feature set g(C^in) as the feature set to be processed by the next 3D-CNN.

When the processing of step S609 is completed, the processing proceeds to FIG. 26. Also, in step S607 of FIG. 25, if it is determined that the guide surface feature amount is not to be input to the next 3D-CNN, the processing proceeds to FIG. 26. In other words, if there is no guide surface feature amount derivation unit immediately before the 3D-CNN, the processing of steps S608 and S609 is omitted. For example, when the encoding device 600 has the configuration example shown in FIG. 24, when the next 3D-CNN is CNN616 (CNN_E1), CNN618 (CNN_E2), or CNN621 (CNN_E4), the processing of steps S608 and S609 is executed, and when the next 3D-CNN is CNN619 (CNN_E3), CNN622 (CNN_E5), or CNN623 (CNN_E6), the processing of steps S608 and S609 is omitted.

In step S621 of FIG. 26, CNN616 (CNN_E1), CNN618 (CNN_E2), CNN619 (CNN_E3), CNN621 (CNN_E4), CNN622 (CNN_E5), or CNN623 (CNN_E6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

In step S622, CNN623 (CNN_E6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., there are unprocessed 3D-CNNs (at least CNN623 (CNN_E6) is unprocessed)), processing proceeds to step S623.

In step S623, CNN616 (CNN_E1), CNN618 (CNN_E2), CNN619 (CNN_E3), CNN621 (CNN_E4), or CNN622 (CNN_E5) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN616 (CNN_E1), CNN618 (CNN_E2), CNN619 (CNN_E3), CNN621 (CNN_E4), or CNN622 (CNN_E5) sets the sparse tensor {C^out, F^out} resulting from the calculation it derived as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or 3D-CNN).

When the processing of step S623 ends, the process returns to step S607 in FIG. 25, and the subsequent processing is executed. In other words, the processing of each of steps S607 to S609 in FIG. 25 and the processing of each of steps S621 to S623 in FIG. 26 are executed for each 3D-CNN (including the guide surface feature derivation unit if one exists immediately before the 3D-CNN).

If it is determined in step S622 of FIG. 26 that processing has been performed for all 3D-CNNs (i.e., processing has been performed for CNN623 (CNN_E6)), processing proceeds to step S624.

In step S624, CNN 623 (CNN_E6) sets {C_Y, F_Y} ← {C^out, F^out}. In other words, the last CNN 623 (CNN_E6) sets the sparse tensor {C^out, F^out}, which is the result of the calculation it derived, as the output sparse tensor {C_Y, F_Y}.

In step S625, the coordinate encoding unit 624 encodes the coordinate set C_Y of the output sparse tensor {C_Y, F_Y} to generate a coordinate bitstream. It also outputs the coordinate bitstream to the outside of the encoding device 600.

In step S626, the feature encoding unit 625 encodes the feature set F_Y of the output sparse tensor {C_Y, F_Y} to generate a feature bitstream. The feature bitstream is then output to the outside of the encoding device 600.

In step S627, the scores for each scale are encoded to generate a score bit stream. The score bit stream is then output to the outside of the encoding device 600.

When the processing of step S627 is completed, the encoding process ends.

<Flow of guide surface feature set derivation process>
Next, an example of the flow of the guide surface feature amount set derivation process executed in step S608 in FIG. 25 will be described with reference to the flowchart in FIG.

When the guide surface feature amount set derivation process is started, in step S631, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 extracts coordinates c_i ∈ C^in. In other words, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 selects one coordinate c_i from the coordinate set C^in to be processed, and makes it the processing target.

In step S632, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 searches for the nearest point p_i on the decoded guide surface from the selected coordinate c_i.

In step S633, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 returns a vector g(c_i) = [p_i - c_i, a_i] that lists the vector and attribute value up to the nearest point found. In other words, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 derives the guide surface feature amount g(c_i) based on the positional relationship between the guide surface and the point (in this example, p_i - c_i) and the attribute value (in this example, a_i).

In step S634, the guide surface feature derivation unit 615, the guide surface feature derivation unit 617, or the guide surface feature derivation unit 620 determines whether or not processing has been performed for all coordinates c_i ∈ C^in. If it is determined that an unprocessed coordinate c_i exists, the process returns to step S631, and the subsequent processes are executed. In other words, the processes of steps S631 to S634 are executed for each coordinate value c_i in the coordinate set C^in to be processed.

If it is determined in step S634 that processing has been performed for all coordinate values c_i, the guide surface feature set derivation process ends and processing returns to FIG. 25.

By performing each process in this manner, the encoding device 600 can encode not only geometry but also attributes.

<Decoding device>
FIG. 28 is a block diagram showing a main configuration example of a decoding device when applying methods 1-6. The decoding device 650 shown in FIG. 28 is a device that decodes a bit stream in which the geometry of a point cloud (3D data) is encoded, similar to the case of the decoding device 350. For example, the decoding device 650 decodes bit streams (guide surface bit stream, coordinate bit stream, feature bit stream, and point number bit stream) generated by encoding the point cloud (geometry and attributes) by the encoding device 600, and generates (restores) the point cloud (geometry and attributes).

Note that FIG. 28 shows the main processing units, data flows, etc., and is not necessarily all that is shown in FIG. 28. In other words, in the decoding device 650, there may be processing units that are not shown as blocks in FIG. 28, and there may be processing and data flows that are not shown as arrows, etc. in FIG. 28.

As shown in FIG. 28, the decoding device 650 has a score decoding unit 661, a coordinate decoding unit 662, a feature decoding unit 663, a guide surface decoding unit 664, a CNN 665 (CNN_D1), a guide surface feature derivation unit 666, a CNN 667 (CNN_D2), an occupancy state classification unit 668, a CNN 669 (CNN_D3), a guide surface feature derivation unit 670, a CNN 671 (CNN_D4), an occupancy state classification unit 672, a CNN 673 (CNN_D5), a guide surface feature derivation unit 674, a CNN 675 (CNN_D6), and an occupancy state classification unit 676.

The score decoding unit 661 is a processing unit similar to the score decoding unit 361 (FIG. 11) and performs similar processing. For example, the score decoding unit 661 may obtain the score bit stream input to the decoding device 650, decode it, and generate (restore) information indicating the scores of the coordinate sets at each scale (N_X, N_X', N_X''). The score decoding unit 661 may supply N_X to the occupancy state classification unit 676, supply N_X' to the occupancy state classification unit 672, and supply N_X'' to the occupancy state classification unit 668.

The coordinate decoding unit 662 is a processing unit similar to the coordinate decoding unit 362 (FIG. 11) and performs similar processing. For example, the coordinate decoding unit 662 may obtain the coordinate bit stream input to the decoding device 650, decode it, and generate (restore) the coordinate set C_Y. The coordinate decoding unit 662 may supply the generated coordinate set C_Y to the CNN 665 (CNN_D1).

The feature decoding unit 663 is a processing unit similar to the feature decoding unit 363 (FIG. 11) and performs similar processing. For example, the feature decoding unit 663 may obtain the feature bit stream input to the decoding device 650, perform entropy decoding, and generate (restore) a feature set F_Y. The feature decoding unit 663 may supply the generated feature set F_Y to the CNN 665 (CNN_D1).

The guide surface decoding unit 664 is a processing unit similar to the guide surface decoding unit 613 (FIG. 24), and executes processing related to the decoding of the guide surface bit stream. For example, the guide surface decoding unit 664 acquires the guide surface bit stream input to the decoding device 650, decodes it, and generates a decoded guide surface including texture. In other words, the guide surface decoding unit 664 decodes the bit stream to generate not only geometry, but also texture coordinates and texture images of each vertex. This decoding method is arbitrary, similar to the case of the guide surface decoding unit 613 (FIG. 24). The guide surface decoding unit 664 may supply the generated decoded guide surface (including texture) to the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, and the guide surface feature amount derivation unit 674.

CNN665 (CNN_D1) is a processing unit similar to CNN365 (CNN_D1) (Figure 11) and performs similar processing. However, in this case, the feature set in the sparse tensor processed by CNN665 (CNN_D1) includes not only geometry information but also attribute information.

The guide surface feature amount derivation unit 666 is a processing unit similar to the guide surface feature amount derivation unit 366 (Figure 11) and performs similar processing. However, in order to use attribute information on the texture for encoding and decoding, the guide surface feature amount derivation unit 666 also includes features extracted from the texture in the guide surface feature amount. Specifically, the guide surface feature amount derivation unit 666 concatenates the texture values a_i (for example, RGB values for color information) at the nearest point p_i on the guide mesh. In other words, the vectors are concatenated to obtain g(c_i) = [p_i - c_i, a_i].

CNN667 (CNN_D2) is a processing unit similar to CNN367 (CNN_D2) (Figure 11) and performs similar processing. However, in this case, the number of dimensions of the guide surface feature increases by the number of dimensions of the attribute value, so it is necessary to increase the input dimensions of CNN667 (CNN_D2).

The occupancy state classification unit 668 is a processing unit similar to the occupancy state classification unit 368 (Figure 11) and performs similar processing. For example, the occupancy state classification unit 668 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X'' coordinate values of the occupancy probability value, delete the other coordinate values, and generate the sparse tensor X'' = {C_X'', F_X''} of the calculation result. The occupancy state classification unit 668 may supply the sparse tensor X'' = {C_X'', F_X''} of the calculation result to the CNN 669 (CNN_D3).

CNN669 (CNN_D3) is a processing unit similar to CNN369 (CNN_D3) (Figure 11) and performs similar processing.

The guide surface feature amount derivation unit 670 is a processing unit similar to the guide surface feature amount derivation unit 370 (FIG. 11) and performs similar processing. However, like the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670 also includes features extracted from the texture in the guide surface feature amount in order to use attribute information on the texture for encoding and decoding.

CNN671 (CNN_D4) is a processing unit similar to CNN371 (CNN_D4) (Figure 11) and performs similar processing. However, in this case, the number of dimensions of the guide surface feature increases by the number of dimensions of the attribute value, so it is necessary to increase the input dimensions of CNN671 (CNN_D4).

The occupancy state classification unit 672 is a processing unit similar to the occupancy state classification unit 372 (Figure 11) and performs similar processing. For example, the occupancy state classification unit 672 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X' coordinate values of the occupancy probability value, delete the other coordinate values, and generate the sparse tensor X' = {C_X', F_X'} of the calculation result. The occupancy state classification unit 672 may supply the sparse tensor X' = {C_X', F_X'} of the calculation result to the CNN 673 (CNN_D5).

CNN673 (CNN_D5) is a processing unit similar to CNN373 (CNN_D5) (Figure 11) and performs similar processing.

The guide surface feature amount derivation unit 674 is a processing unit similar to the guide surface feature amount derivation unit 374 (Figure 11) and performs similar processing. However, like the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 674 also includes features extracted from the texture in the guide surface feature amount in order to use attribute information on the texture for encoding and decoding.

CNN675 (CNN_D6) is a processing unit similar to CNN375 (CNN_D6) (Figure 11) and performs similar processing. However, in this case, the number of dimensions of the guide surface features increases by the number of dimensions of the attribute values, so it is necessary to increase the input dimensions of CNN675 (CNN_D6). Furthermore, CNN675 (CNN_D6) needs to increase the output dimensions by the number of attribute values.

The occupancy state classification unit 676 is a processing unit similar to the occupancy state classification unit 376 (FIG. 11) and performs similar processing. For example, the occupancy state classification unit 676 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X coordinate values of the occupancy probability value, delete the other coordinate values, and generate the sparse tensor X = {C_X, F_X} of the calculation result. The occupancy state classification unit 676 may output the decoded point group C_X of the calculation result to the outside of the decoding device 650. The occupancy state classification unit 676 may also output the attributes of the calculation result to the outside of the decoding device 650.

For example, if the attribute is an RGB value, each feature vector f_i ∈ F_X in the feature vector set F_X output by the CNN 675 (CNN_D6) is expanded by three dimensions to become an RGB value. The occupancy state classification unit 676 sets the attribute value as the attribute of each point in the decoded point group C_X and outputs it.

With this configuration, the decoding device 650 can decode not only geometry but also attributes.

<Decryption process flow>
An example of the flow of the decoding process executed by the decoding device 650 will be described with reference to the flowcharts of FIGS.

In this case, when the decoding process starts, the score decoding unit 661 decodes the score bit stream in step S651 of FIG. 29, and generates scores for each scale (N_X, N_X', N_X'').

In step S652, the coordinate decoding unit 662 decodes the coordinate bit stream and generates a coordinate set C_Y.

In step S653, the feature decoding unit 663 decodes the feature bit stream and generates a feature set F_Y.

In step S654, the guide surface decoding unit 664 decodes the guide surface bitstream and generates a decoded guide surface and decoded texture data.

In step S655, CNN665 (CNN_D1) sets {C^in, F^in} ← {C_Y, F_Y}. In other words, CNN665 (CNN_D1) sets the sparse tensor Y = {C_Y, F_Y} as the sparse tensor {C^in, F^in} to be processed.

In step S656, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) determines whether or not to input guide surface features to its own 3D-CNN. If it is determined that the guide surface features are to be input, the process proceeds to step S657. In other words, if the CNN to be processed is CNN667 (CNN_D2), CNN671 (CNN_D4), or CNN375 (CNN_D6), the process proceeds to step S657.

In step S657, the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, or the guide surface feature amount derivation unit 674 executes a guide surface feature amount set derivation process to derive a guide surface feature amount set g(C^in). This guide surface feature amount set derivation process is executed in the same manner as described with reference to the flowchart in FIG. 27. In other words, the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, or the guide surface feature amount derivation unit 674 derives the guide surface feature amount set g(C^in) using the sparse tensor to be processed, the decoded guide surface, and the decoded texture data.

In step S658, the guide surface feature derivation unit 666, the guide surface feature derivation unit 670, or the guide surface feature derivation unit 674 sets F^in ← [F^in, g(C^in)]. In other words, the guide surface feature derivation unit 766, the guide surface feature derivation unit 670, or the guide surface feature derivation unit 674 sets the concatenation of the feature set to be processed and the guide surface feature set g(C^in) as the feature set to be processed by the next 3D-CNN.

When the processing of step S658 ends, the process proceeds to FIG. 30. Also, if it is determined in step S656 of FIG. 29 that guide surface features are not to be input to its own 3D-CNN, the processing of steps S657 and S658 is omitted, and the process proceeds to FIG. 30. In other words, if the CNN to be processed is CNN665 (CNN_D1), CNN669 (CNN_D3), or CNN673 (CNN_D5), the processing of steps S657 and S658 is omitted, and the process proceeds to FIG. 30.

In step S671 of FIG. 30, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

In step S672, the occupancy state classification unit 668, the occupancy state classification unit 672, or the occupancy state classification unit 676 determines whether or not to classify the occupancy state. If it is determined that the occupancy state is to be classified, the process proceeds to step S673. In other words, if the process of step S671 is executed by CNN 667 (CNN_D2), CNN 671 (CNN_D4), or CNN 675 (CNN_D6), the process proceeds to step S673.

In step S673, the occupancy state classification unit 668, occupancy state classification unit 672, or occupancy state classification unit 676 predicts the occupancy probability of each coordinate in the coordinate set C^out, extracts the top k coordinates with the highest occupancy probability, and deletes the remaining coordinates. At this time, the scale value corresponding to the sparse tensor to be processed, among N_X, N_X', or N_X'', is applied to this k. In addition, the occupancy state classification unit 668, occupancy state classification unit 672, or occupancy state classification unit 676 also deletes the features corresponding to the deleted coordinates in F^out.

When the process of step S673 ends, the process proceeds to step S674. Also, if it is determined in step S672 that occupancy state classification is not to be performed, the process of step S673 is omitted and the process proceeds to step S674. In other words, if the process of step S671 is performed by CNN665 (CNN_D1), CNN669 (CNN_D3), or CNN673 (CNN_D5), the process of step S673 is omitted and the process proceeds to step S674.

In step S674, CNN675 (CNN_D6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., there are unprocessed 3D-CNNs (at least CNN675 (CNN_D6) is unprocessed)), processing proceeds to step S675.

In step S675, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) sets the sparse tensor {C^out, F^out} of the calculation result derived by itself as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or occupancy state classification unit). When the processing of step S675 is completed, the process returns to step S656 in FIG. 29, and subsequent processing is executed. That is, the processes of steps S656 to S658 in FIG. 29 and steps S671 to S675 in FIG. 30 are executed for each 3D-CNN (which may include a guide surface feature derivation unit and an occupancy state classification unit).

If it is determined in step S674 of FIG. 30 that processing has been performed for all 3D-CNNs, processing proceeds to step S676.

In step S676, the occupancy state classification unit 676 sets C_X ← C^out. In other words, the occupancy state classification unit 676 sets the coordinate set C^out of the calculation result as the decoding result (decoded point group C_X). The occupancy state classification unit 676 also extracts attribute values from F^out. For example, if the attribute is an RGB value, the occupancy state classification unit 676 extracts the last three dimensions of each feature vector of F^out and sets this vector as the attribute.

In step S677, the occupancy state classification unit 676 outputs the obtained decoded point group C_X to the outside of the decoding device 650. In addition, the occupancy state classification unit 676 outputs the obtained attributes to the outside of the decoding device 650.

When step S677 is completed, the decoding process ends.

By performing each process in this manner, the decoding device 350 can decode not only geometry but also attributes.

<Comparison>
FIG. 31 shows an example of a comparison result of RD costs between the case where the present technology described above is applied and the case where the conventional method is applied.

The graph in Figure 31 shows an example of an RD curve plot when a point cloud (one frame, approximately 900,000 points, 10-bit accuracy) obtained by scanning a specific person is intra-encoded and intra-decoded.

In the graph of Figure 31, the vertical axis shows mseF PSNR (p2point), or the so-called D1 Metric PSNR. In other words, if the position on the horizontal axis is the same, the higher the position on the vertical axis, the higher the quality of the decoded result. The horizontal axis shows the bit size per point (bpp (bits per point)). In other words, if the position on the vertical axis is the same, the further to the left the position on the horizontal axis, the less code there is.

The solid line in the graph of Figure 31 is an RD curve showing the relationship between bit size and PSNR when encoding a point cloud using a conventional method without using a guide surface. The dotted line in the graph of Figure 31 is an RD curve showing the relationship between bit size and PSNR (Peak Signal-to-Noise Ratio) when encoding a point cloud using a guide surface by applying this technology.

As shown in Figure 31, the dotted lines are generally positioned above the solid lines. Therefore, encoding using a method that applies this technology (a method that uses a guide surface) can obtain better results (higher quality and lower code volume) than conventional methods (methods that do not use a guide surface). In other words, by applying this technology, it is possible to suppress a decrease in encoding efficiency.

<4. Notes>
<Computer>
The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, etc., capable of executing various functions by installing various programs.

FIG. 32 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program.

In the computer 900 shown in FIG. 32, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

Also connected to the bus 904 is an input/output interface 910. An input unit 911, an output unit 912, a memory unit 913, a communication unit 914, and a drive 915 are connected to the input/output interface 910.

The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In a computer configured as described above, the CPU 901 loads a program stored in the storage unit 913, for example, into the RAM 903 via the input/output interface 910 and the bus 904, and executes the program, thereby carrying out the above-mentioned series of processes. The RAM 903 also stores data necessary for the CPU 901 to execute various processes, as appropriate.

The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in the storage unit 913 via the input/output interface 910 by inserting the removable media 921 into the drive 915.

This program can also be provided via a wired or wireless transmission medium, such as a local area network, the Internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

In addition, this program can also be pre-installed in the ROM 902 or memory unit 913.

<Applicable subjects of this technology>
The present technology may be applied to any configuration, for example, to various electronic devices.

Furthermore, for example, the present technology can be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

Furthermore, for example, the present technology can also be applied to a network system consisting of multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video) related services to any terminal, such as a computer, AV (Audio Visual) equipment, portable information processing terminal, or IoT (Internet of Things) device.

In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

<Fields and applications where this technology can be applied>
Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications are also arbitrary.

This technology can be used, for example, as a technology (so-called smart construction) aimed at improving productivity and safety at construction sites and resolving labor shortages, to create digital twins of construction sites through 3D surveying and use them for construction management. For example, this technology can be applied to 3D surveying using sensors mounted on drones or construction machinery, and feedback based on 3D data (e.g. point clouds) obtained by the surveying (e.g. construction progress management and soil volume management).

Such surveys are generally carried out multiple times at different times (e.g., every other day). In other words, point cloud data obtained by surveys at different times is accumulated. Therefore, as the number of surveys increases, the increase in storage capacity and transmission costs for point cloud data sequences can become a problem. There is redundancy between such point clouds from different times, such as having similar structures. In other words, there is temporal redundancy. It is expected that the application of this technology will be effective in compressing point clouds by utilizing redundancy in the temporal direction, i.e., compressing point cloud sequences.

＜Other＞
In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Thus, the value that this "flag" can take may be, for example, two values, 1/0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. In addition, it is assumed that identification information (including flags) is not only included in the bit stream, but also includes difference information of the identification information with respect to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information with respect to the reference information.

Furthermore, various information (metadata, etc.) related to the encoded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the encoded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. In other words, data that are associated with each other may be combined into one piece of data, or each piece of data may be individual data. For example, information associated with encoded data (image) may be transmitted on a transmission path different from that of the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a recording medium different from that of the encoded data (image) (or on a different recording area of the same recording medium). Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.

In this specification, terms such as "composite," "multiplex," "append," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

Furthermore, the embodiments of this technology are not limited to the above-mentioned embodiments, and various modifications are possible without departing from the spirit of this technology.

For example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

Also, for example, the above-mentioned program may be executed in any device. In that case, it is sufficient that the device has the necessary functions (functional blocks, etc.) and is capable of obtaining the necessary information.

Also, for example, each step of a single flowchart may be executed by a single device, or may be shared among multiple devices. Furthermore, if a single step includes multiple processes, those multiple processes may be executed by a single device, or may be shared among multiple devices. In other words, multiple processes included in a single step may be executed as multiple step processes. Conversely, processes described as multiple steps may be executed collectively as a single step.

Furthermore, for example, the processing of the steps describing a program executed by a computer may be executed chronologically in the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, as long as no contradiction arises, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps describing this program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

Furthermore, for example, the multiple technologies related to the present technology can be implemented independently and individually, so long as no contradictions arise. Of course, any multiple of the present technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the present technologies described above can be implemented in combination with other technologies not described above.

The present technology can also be configured as follows.
(1) a guide surface feature amount derivation unit that derives guide surface feature amounts, which are feature amounts based on a positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature amount set, which is a set of feature amounts of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space;
a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature amount set that reflects the guide surface feature amount, and derives a second coordinate set and a second feature amount set;
a coordinate encoding unit that encodes the derived second coordinate set;
a feature encoding unit that encodes the derived second feature set;
an amount of code for the encoded data of the second coordinate set and the second feature set is smaller than an amount of code for the encoded data of the first coordinate set and the first feature set.
(2) The information processing device according to (1), in which a scale of the second coordinate set and the second feature amount set is smaller than a scale of the first coordinate set and the first feature amount set.
(3) The information processing device according to (2), wherein the calculation unit derives the second coordinate set and the second feature amount set by recursively repeating the predetermined calculation a number of times.
(4) The information processing device according to (3), wherein the calculation unit downscales an input coordinate set and a feature amount set in all or a part of the plurality of times of the predetermined calculation.
(5) The information processing device according to (4), wherein the guide surface feature amount derivation unit derives the guide surface feature amount for each of the coordinate set and the feature amount set at some scales or all scales.
(6) a guide surface generation unit that generates the guide surface by using the point cloud;
a guide surface encoding unit that encodes the guide surface and generates encoded data of the guide surface;
The information processing device according to any one of (1) to (5), further comprising: a guide surface decoding unit that decodes encoded data of the guide surface and generates the guide surface.
(7) The information processing device according to any one of (1) to (6), wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with respect to the point.
(8) The information processing device according to any one of (1) to (7), wherein the guide surface is a mesh.
(9) The information processing device according to any one of (1) to (7), wherein the guide surface is a triangle soup.
(10) The information processing device according to any one of (1) to (7), wherein the guide surface is an implicit function.
(11) The information processing device according to any one of (1) to (10), further comprising: a parameter set selection unit that selects a parameter set to be applied to encoding the point cloud.
(12) a guide surface generation unit that generates the guide surface by using the point cloud;
The information processing device according to (11), further comprising: a guide surface encoding unit that applies the selected parameter set to encode the guide surface and generates encoded data of the guide surface.
(13) The information processing device according to any one of (1) to (12), wherein the guide surface feature amount derivation unit derives the guide surface feature amount by using the coordinate set, the feature amount set including attribute information, and the guide surface including a texture.
(14) deriving a guide surface feature amount, which is a feature amount based on a positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points forming the point cloud, a first feature amount set, which is a set of feature amounts of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space;
performing a calculation using the first coordinate set and the first feature amount set reflecting the guide surface feature amount to derive a second coordinate set and a second feature amount set;
encoding the derived second set of coordinates;
encoding the derived second feature set.

(21) A coordinate decoding unit that decodes encoded data to generate a first coordinate set that is a set of coordinates of points that form the point cloud;
a feature decoding unit that decodes the encoded data to generate a first feature set that is a set of features of the coordinates;
a guide surface feature amount derivation unit that derives a guide surface feature amount, which is a feature amount based on a positional relationship between the guide surface and the point, by using the generated first coordinate set, the generated first feature amount set, and a guide surface, which is a reference surface formed in a three-dimensional space;
a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature amount set that reflects the guide surface feature amount, and derives a second coordinate set and a second feature amount set,
an amount of code for encoded data of the first coordinate set and the first feature set is smaller than an amount of code for encoded data of the second coordinate set and the second feature set.
(22) The information processing device according to (21), in which a scale of the second coordinate set and the second feature amount set is larger than a scale of the first coordinate set and the first feature amount set.
(23) The information processing device according to (22), wherein the calculation unit derives the second coordinate set and the second feature amount set by recursively repeating the predetermined calculation a number of times.
(24) The information processing device according to (23), wherein the calculation unit upscales a set of coordinates and a set of feature amounts that are input in all or a part of the plurality of times of the predetermined calculation.
(25) The information processing device according to (24), wherein the guide surface feature amount derivation unit derives the guide surface feature amount for each of the coordinate set and the feature amount set at some scales or all scales.
(26) The information processing device according to any one of (21) to (25), further comprising: a guide surface decoding unit that decodes encoded data of the guide surface and generates the guide surface.
(27) The information processing device according to any one of (21) to (26), wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with respect to the point.
(28) The information processing device according to any one of (21) to (27), wherein the guide surface is a mesh.
(29) The information processing device according to (28), further comprising: a fitting unit that fits the guide surface to the point cloud.
(30) The information processing device according to any one of (21) to (27), wherein the guide surface is a triangle soup.
(31) The information processing device according to any one of (21) to (27), wherein the guide surface is an implicit function.
(32) The information processing device according to any one of (21) to (31), wherein the guide surface feature amount derivation unit derives the guide surface feature amount by using the coordinate set, the feature amount set including attribute information, and the guide surface including a texture.
(33) Decoding the encoded data to generate a first set of coordinates that is a set of coordinates of points that form the point cloud;
Decoding the encoded data to generate a first feature set that is a set of features of the coordinates;
deriving a guide surface feature amount, which is a feature amount based on a positional relationship between the guide surface and the point, using the generated first coordinate set, the generated first feature amount set, and a guide surface, which is a reference surface formed in a three-dimensional space;
performing a predetermined calculation using the first coordinate set and the first feature amount set reflecting the guide surface feature amount to derive a second coordinate set and a second feature amount set;
an amount of code for the encoded data of the first coordinate set and the first feature set is smaller than an amount of code for the encoded data of the second coordinate set and the second feature set.

100 Calculator, 101 Guide surface feature derivation unit, 102 CNN, 120 Calculator, 121 Guide surface feature derivation unit, 122 Calculation unit, 300 Encoding device, 311 Guide surface generation unit, 312 Guide surface encoding unit, 313 Guide surface decoding unit, 314 Sparse tensor construction unit, 315 Guide surface feature derivation unit, 316 CNN (CNN_E1), 317 Guide surface feature derivation unit, 318 CNN (CNN_E2), 319 CNN (CNN_E3), 320 Guide surface feature derivation unit, 321 CNN (CNN_E4), 322 CNN (CNN_E5), 323 CNN (CNN_E6), 324 Coordinate encoding unit, 325 feature encoding unit, 326 score encoding unit, 350 decoding device, 361 score decoding unit, 362 coordinate decoding unit, 363 feature decoding unit, 364 guide surface decoding unit, 365 CNN (CNN_D1), 366 guide surface feature derivation unit, 367 CNN (CNN_D2), 368 occupancy state classification unit, 369 CNN (CNN_D3), 370 guide surface feature derivation unit, 371 CNN (CNN_D4), 372 occupancy state classification unit, 373 CNN (CNN_D5), 374 guide surface feature derivation unit, 375 CNN (CNN_D6), 376 occupancy state classification unit, 900 computer

Claims (20)

a guide surface feature amount derivation unit that derives guide surface feature amounts, which are feature amounts based on a positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points forming the point cloud, a first feature amount set, which is a set of feature amounts of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space;
a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature amount set that reflects the guide surface feature amount, and derives a second coordinate set and a second feature amount set;
a coordinate encoding unit that encodes the derived second coordinate set;
a feature encoding unit that encodes the derived second feature set;
an amount of code for the encoded data of the second coordinate set and the second feature set is smaller than an amount of code for the encoded data of the first coordinate set and the first feature set. the calculation unit recursively repeats the predetermined calculation a plurality of times, and downscales the input coordinate set and feature amount set in all or a portion of the predetermined calculation performed a plurality of times to derive the second coordinate set and the second feature amount set having a smaller scale than the first coordinate set and the first feature amount set;
The information processing device according to claim 1 , wherein the guide surface feature amount derivation unit derives the guide surface feature amount for each of the coordinate set and the feature amount set at some or all of the scales. a guide surface generation unit that generates the guide surface using the point cloud;
a guide surface encoding unit that encodes the guide surface and generates encoded data of the guide surface;
The information processing device according to claim 1 , further comprising: a guide surface decoding unit that decodes encoded data of the guide surface and generates the guide surface. The information processing apparatus according to claim 1 , wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with respect to the point as a reference. The information processing device according to claim 1 , wherein the guide surface is a mesh. The information processing device according to claim 1 , wherein the guide surface is a triangle soup. The information processing apparatus according to claim 1 , wherein the guide surface is an implicit function. The information processing device according to claim 1 , further comprising a parameter set selection unit that selects a parameter set to be applied to encoding the point cloud. The information processing device according to claim 1 , wherein the guide surface feature amount derivation unit derives the guide surface feature amount by using the coordinate set, the feature amount set including attribute information, and the guide surface including a texture. deriving a guide surface feature amount, which is a feature amount based on a positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points forming the point cloud, a first feature amount set, which is a set of feature amounts of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space;
performing a predetermined calculation using the first coordinate set and the first feature amount set reflecting the guide surface feature amount to derive a second coordinate set and a second feature amount set;
encoding the derived second set of coordinates;
encoding the derived second set of features;
a code amount of the encoded data of the second coordinate set and the second feature set is smaller than a code amount of the encoded data of the first coordinate set and the first feature set. a coordinate decoding unit that decodes the encoded data to generate a first set of coordinates that is a set of coordinates of points that form the point cloud;
a feature decoding unit that decodes the encoded data to generate a first feature set that is a set of features of the coordinates;
a guide surface feature amount derivation unit that derives a guide surface feature amount, which is a feature amount based on a positional relationship between the guide surface and the point, by using the generated first coordinate set, the generated first feature amount set, and a guide surface, which is a reference surface formed in a three-dimensional space;
a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature amount set that reflects the guide surface feature amount, and derives a second coordinate set and a second feature amount set,
an amount of code for the encoded data of the first coordinate set and the first feature set is smaller than an amount of code for the encoded data of the second coordinate set and the second feature set. the calculation unit recursively repeats the predetermined calculation a plurality of times, and upscales the input coordinate set and feature amount set in all or a portion of the predetermined calculation performed a plurality of times to derive the second coordinate set and the second feature amount set having a larger scale than the first coordinate set and the first feature amount set;
The information processing device according to claim 11 , wherein the guide surface feature amount derivation unit derives the guide surface feature amount for each of the coordinate set and a part of scales or all of the scales of the feature amount set. The information processing device according to claim 11 , further comprising: a guide surface decoding unit that decodes encoded data of the guide surface and generates the guide surface. The information processing apparatus according to claim 11 , wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with respect to the point. The information processing device according to claim 11 , wherein the guide surface is a mesh. The information processing device according to claim 11 , further comprising a fitting unit that fits the guide surface to the point cloud. The information processing device according to claim 11 , wherein the guide surface is a triangle soup. The information processing device according to claim 11 , wherein the guide surface is an implicit function. The information processing device according to claim 11 , wherein the guide surface feature amount derivation unit derives the guide surface feature amount by using the coordinate set, the feature amount set including attribute information, and the guide surface including a texture. Decoding the encoded data to generate a first set of coordinates that are a set of coordinates of points that form the point cloud;
Decoding the encoded data to generate a first feature set that is a set of features of the coordinates;
Using the generated first coordinate set, the generated first feature amount set, and a guide surface that is a reference surface formed in a three-dimensional space, a guide surface feature amount that is a feature amount based on a positional relationship between the guide surface and the point is derived;
performing a predetermined calculation using the first coordinate set and the first feature amount set reflecting the guide surface feature amount to derive a second coordinate set and a second feature amount set;
an amount of code for the encoded data of the first coordinate set and the first feature set is smaller than an amount of code for the encoded data of the second coordinate set and the second feature set.

Images