WO2025205259A1 - Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device - Google Patents

Abstract

A three-dimensional data encoding method according to the present invention selects any one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method that is different from the first encoding method (S601), when the first encoding method is selected (first encoding method in S602), one or more first components are encoded using the first encoding method to generate first encoded data, (S603), when the second encoding method is selected (second encoding method in S602), the one or more first components are encoded using the second encoding method to generate second encoded data (S604), first information indicating the encoding method that was selected is stored in first metadata in which metadata of the first encoding method is extended (S605), and a bitstream including the first metadata and the first encoded data or the second encoded data is generated (S606).

Description

Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

This disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device.

In the future, devices and services that utilize 3D data are expected to become more widespread in a wide range of fields, including computer vision for autonomous vehicle or robot operation, map information, surveillance, infrastructure inspection, and video distribution. 3D data can be acquired in a variety of ways, including distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras.

One method of representing three-dimensional data is a point cloud, which uses a group of points in three-dimensional space to represent the shape of a three-dimensional structure. A point cloud stores the position and color of the points. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds contain a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the data volume through encoding, just as with two-dimensional video images (examples include MPEG-4 AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding), which are standardized by MPEG).

In addition, point cloud compression is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing.

In addition, technology is known that uses three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

International Publication No. 2014/020663

When encoding and decoding such three-dimensional data, it is desirable to be able to improve the encoding efficiency.

The purpose of this disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method for encoding three-dimensional data including one or more first components, the method selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method, and if the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method, and if the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method, storing first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method, and generating a bitstream including the first metadata and the first encoded data or the second encoded data.

A three-dimensional data decoding method according to one aspect of the present disclosure is a three-dimensional data decoding method for decoding three-dimensional data including one or more first components, the method comprising: obtaining a bitstream including first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data; obtaining, from the first metadata, first information indicating an encoding method used to encode the one or more first components, from a plurality of encoding methods including the first encoding method and a second encoding method different from the first encoding method; and, if the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method; and, if the first information indicates the second encoding method, generating the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

This disclosure provides a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency.

FIG. 1 is a diagram illustrating an example of the configuration of a three-dimensional data encoding/decoding system according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of point cloud data according to the embodiment. FIG. 3 is a diagram showing an example of the structure of a data file in which point cloud data information according to the embodiment is described. FIG. 4 is a diagram illustrating an example of the configuration of mesh data according to the embodiment. FIG. 5 is a diagram showing an example of the structure of a data file in which information on mesh data according to the embodiment is described. FIG. 6 is a diagram showing types of three-dimensional data according to the embodiment. FIG. 7 is a diagram illustrating a configuration of a first encoding unit according to the embodiment. FIG. 8 is a block diagram of a first encoding unit according to the embodiment. FIG. 9 is a diagram illustrating a configuration of a first decoding unit according to the embodiment. FIG. 10 is a block diagram of a first decoding unit according to the embodiment. FIG. 11 is a diagram illustrating a configuration of a second encoding unit according to the embodiment. FIG. 12 is a block diagram of a second encoding unit according to the embodiment. FIG. 13 is a diagram illustrating a configuration of a second decoding unit according to the embodiment. FIG. 14 is a block diagram of a second decoding unit according to the embodiment. FIG. 15 is a block diagram of a position information encoding unit according to an embodiment. FIG. 16 is a block diagram of a position information decoding unit according to the embodiment. FIG. 17 is a block diagram of an octree encoding unit according to an embodiment. FIG. 18 is a diagram illustrating an example of location information according to the embodiment. FIG. 19 is a diagram showing an example of an octree representation of position information according to an embodiment. FIG. 20 is a block diagram of an octree decoding unit according to an embodiment. FIG. 21 is a block diagram of an attribute information encoding unit according to an embodiment. FIG. 22 is a block diagram of an attribute information decoding unit according to the embodiment. FIG. 23 is a block diagram of an attribute information encoding unit according to an embodiment. FIG. 24 is a block diagram of an attribute information decoding unit according to the embodiment. FIG. 25 is a block diagram of a three-dimensional data encoding device according to an embodiment. FIG. 26 is a block diagram of a three-dimensional data decoding device according to an embodiment. FIG. 27 is a diagram showing the relationship between tiles and slices according to the embodiment. FIG. 28 is a diagram illustrating an example of the configuration of a bitstream according to the embodiment. FIG. 29 is a block diagram showing an example of the configuration of a three-dimensional data encoding device according to an embodiment. FIG. 30 is a diagram showing an example of the structure of coded data and NAL units according to an embodiment. FIG. 31 is a diagram illustrating an example of the semantics of pcc_nal_unit_type according to an embodiment. FIG. 32 is a diagram showing a part of the processing in the three-dimensional data generation system according to the embodiment. FIG. 33 is a diagram illustrating components of Gaussian data according to the embodiment. FIG. 34 is a diagram illustrating a rendering process of Gaussian data according to an embodiment. FIG. 35 is a diagram illustrating an example of the number of SH coefficients of spherical harmonic functions according to the embodiment. FIG. 36 is a diagram illustrating an example of input and output of a spherical harmonic function according to an embodiment. FIG. 37 is a diagram illustrating a rendering process according to an embodiment. FIG. 38 is a block diagram of an encoding device according to an embodiment. FIG. 39 is a block diagram of a decoding device according to an embodiment. FIG. 40 is a block diagram of an encoding device according to an embodiment. FIG. 41 is a block diagram of a decoding device according to an embodiment. FIG. 42 is a diagram illustrating an example of mapping of Gaussian data according to an embodiment. FIG. 43 is a diagram illustrating an example of mapping information according to an embodiment. FIG. 44 is a diagram illustrating an example of the configuration of conversion information according to the embodiment. FIG. 45 is a diagram illustrating an example of the structure of coded data according to an embodiment. FIG. 46 is a diagram showing an example of the G-PCC attribute type of Gaussian data according to the embodiment. FIG. 47 is a flowchart of processing by the decoding device according to the embodiment. FIG. 48 is a block diagram illustrating a system for encoding and decoding first Gaussian data according to an embodiment. FIG. 49 is a block diagram of a second pre-processing unit according to an embodiment. FIG. 50 is a diagram illustrating a conversion process according to an embodiment. FIG. 51 is a diagram showing a protocol stack of an encoding standard for encoding Gaussian data according to an embodiment. FIG. 52 is a block diagram of an encoding device according to an embodiment. FIG. 53 is a block diagram of a decoding device according to an embodiment. FIG. 54 is a flowchart of the encoding process according to the embodiment. FIG. 55 is a flowchart of the decoding process according to the embodiment. FIG. 56 is a diagram showing the basic data structure of a Gaussian codec according to an embodiment. FIG. 57 is a diagram showing an example of the structure of a 3D data unit according to an embodiment. FIG. 58 is a diagram showing an example of the configuration of a 3DPS according to an embodiment. FIG. 59 is a diagram showing an example of the configuration of Gaussian data component information according to the embodiment. Figure 60 is a diagram showing an example of the configuration of G-PCC mapping information relating to an embodiment. FIG. 61 is a diagram showing an example of the configuration of V3C mapping information according to an embodiment. FIG. 62 is a diagram illustrating an example of the configuration of a Gaussian data unit according to an embodiment. FIG. 63 is a diagram illustrating a protocol stack for a standard for encoding 3D Gaussians according to an embodiment. FIG. 64 is a diagram showing a protocol stack of a standard for encoding a 3D model according to an embodiment. FIG. 65 is a diagram illustrating an example of the configuration of Gaussian component information according to the embodiment. FIG. 66 is a diagram showing a protocol stack of an encoding standard for encoding Gaussian data according to an embodiment. FIG. 67 is a block diagram of an encoding device according to an embodiment. FIG. 68 is a block diagram of a decoding device according to an embodiment. FIG. 69 is a flowchart of the encoding process according to the embodiment. FIG. 70 is a flowchart of the decoding process according to the embodiment. FIG. 71 is a diagram showing an example of the configuration of an extended V3C unit according to an embodiment. FIG. 72 is a diagram illustrating an example of the configuration of a V3C unit according to an embodiment. Figure 73 is a diagram showing an example configuration of a GCL NAL unit relating to an embodiment. FIG. 74 is a diagram showing an example of the configuration of an extended V3C parameter set according to an embodiment. FIG. 75 is a diagram showing an example of the configuration of Gaussian data component information according to the embodiment. FIG. 76 is a diagram showing an example of the configuration of video mapping information according to the embodiment. FIG. 77 is a diagram illustrating an example of the configuration of 3D mapping information according to the embodiment. FIG. 78 is a diagram showing an extended V3C protocol stack according to an embodiment. FIG. 79 is a diagram showing an extended V3C protocol stack according to an embodiment. FIG. 80 is a flowchart of the encoding process performed by the encoding device according to the embodiment. FIG. 81 is a block diagram of an encoding device according to an embodiment. FIG. 82 is a flowchart of the decoding process by the decoding device according to the embodiment. FIG. 83 is a block diagram of a decoding device according to an embodiment.

A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method for encoding three-dimensional data including one or more first components, the method selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method, and if the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method, and if the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method, storing first information indicating the selected encoding method in first metadata that is an extension of the metadata of the first encoding method, and generating a bitstream including the first metadata and the first encoded data or the second encoded data.

This allows the three-dimensional data encoding method to selectively select from multiple encoding methods the encoding method to be used for encoding three-dimensional data. This has the potential to improve encoding efficiency. Furthermore, by extending the video-based three-dimensional data encoding method, it is possible to easily build a system that can achieve the above processing.

For example, the three-dimensional data encoding method may further convert the one or more first components into one or more second components corresponding to the selected encoding method, and if the first encoding method is selected, generate the first encoded data by encoding the one or more second components using the first encoding method, and if the second encoding method is selected, generate the second encoded data by encoding the one or more second components using the second encoding method, and the first metadata may further include second information indicating a correspondence between the one or more first components and the one or more second components.

According to this, the three-dimensional data encoding method can encode three-dimensional data using the selected encoding method by converting the data into a format that corresponds to the selected encoding method. Furthermore, the decoding device can convert multiple second components into multiple first components using second information included in the bitstream.

For example, the second encoding method may be a geometry-based three-dimensional data encoding method. For example, the second encoding method may be a neural network-based three-dimensional data encoding method. For example, the second encoding method may be a three-dimensional data encoding method using AI. For example, the three-dimensional data may be Gaussian data.

This allows the three-dimensional data encoding method to selectively select the encoding method to use for encoding Gaussian data from multiple encoding methods, potentially improving encoding efficiency.

A three-dimensional data decoding method according to one aspect of the present disclosure is a three-dimensional data decoding method for decoding three-dimensional data including one or more first components, the method comprising: obtaining a bitstream including first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method, and encoded data; obtaining, from the first metadata, first information indicating an encoding method used to encode the one or more first components, from a plurality of encoding methods including the first encoding method and a second encoding method different from the first encoding method; and, if the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method; and, if the first information indicates the second encoding method, generating the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.

This allows the encoding device to selectively select an encoding method to use for encoding three-dimensional data from multiple encoding methods, potentially improving encoding efficiency. Furthermore, the three-dimensional data decoding method can appropriately decode encoded data generated by the encoding device. Furthermore, by extending the video-based three-dimensional data encoding method, a system capable of easily implementing the above processing can be constructed.

For example, the encoded data may be generated by encoding one or more second components resulting from the conversion of the one or more first components, and the three-dimensional data decoding method may further include: acquiring, from the first metadata, second information indicating a correspondence between the one or more first components and the one or more second components; if the first information indicates the first encoding method, generating the one or more second components by decoding the encoded data using the first decoding method; if the first information indicates the second encoding method, generating the one or more second components by decoding the encoded data using the second decoding method; and converting the one or more second components into the one or more first components using the correspondence indicated in the second information.

This allows the encoding device to encode three-dimensional data using the selected encoding method by converting the data into a format compatible with the selected encoding method. Furthermore, the three-dimensional data decoding device can convert multiple second components into multiple first components using second information included in the bitstream.

For example, the second encoding method may be a geometry-based three-dimensional data encoding method. For example, the second encoding method may be a neural network-based three-dimensional data encoding method. For example, the second encoding method may be a three-dimensional data encoding method using AI.

For example, the three-dimensional data may be Gaussian data.

This allows the encoding device to selectively select from multiple encoding methods the encoding method to use for encoding Gaussian data. This has the potential to improve encoding efficiency.

A three-dimensional data encoding device according to one aspect of the present disclosure is a three-dimensional data encoding device that encodes three-dimensional data including one or more first components, and includes a processor and a memory. The processor uses the memory to select one of a plurality of encoding methods, including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method different from the first encoding method. If the first encoding method is selected, the processor generates first encoded data by encoding the one or more first components using the first encoding method. If the second encoding method is selected, the processor generates second encoded data by encoding the one or more first components using the second encoding method. The processor stores first information indicating the selected encoding method in first metadata that is an extension of the metadata for the first encoding method. The processor generates a bitstream that includes the first metadata and the first encoded data or the second encoded data.

This allows the three-dimensional data encoding device to selectively select from multiple encoding methods the encoding method to be used for encoding three-dimensional data. This has the potential to improve encoding efficiency. Furthermore, by expanding the video-based three-dimensional data encoding method, it is possible to easily build a system that can achieve the above processing.

A three-dimensional data decoding device according to one aspect of the present disclosure is a three-dimensional data decoding device that decodes three-dimensional data including one or more first components, and includes a processor and a memory. The processor uses the memory to obtain a bitstream that includes encoded data and first metadata, which is an extension of the metadata of a first encoding method that is a video-based three-dimensional data encoding method. The processor obtains, from the first metadata, first information indicating which encoding method, out of multiple encoding methods including the first encoding method and a second encoding method, was used to encode the one or more first components. If the first information indicates the first encoding method, the encoded data is decoded using a first decoding method corresponding to the first encoding method to generate the one or more first components. If the first information indicates the second encoding method, the encoded data is decoded using a second decoding method corresponding to the second encoding method to generate the one or more first components.

This allows the encoding device to selectively select from multiple encoding methods the encoding method to use for encoding 3D data. This potentially improves encoding efficiency. Furthermore, the 3D data decoding device can properly decode the encoded data generated by the encoding device. Furthermore, by expanding the video-based 3D data encoding method, a system capable of easily implementing the above processing can be constructed.

Furthermore, these comprehensive or specific aspects may be realized as a system, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

The following embodiments are described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims are described as optional components.

(Embodiment)
[Three-dimensional data encoding/decoding system]
First, the configuration of a three-dimensional data encoding/decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding/decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding/decoding system includes a three-dimensional data encoding system 101, a three-dimensional data decoding system 102, a sensor terminal 103, and an external connection unit 104.

The three-dimensional data encoding system 101 generates encoded data or multiplexed data by encoding three-dimensional data such as three-dimensional point cloud data or three-dimensional mesh data. The three-dimensional data encoding system 101 may be a three-dimensional data encoding device implemented by a single device, or a system implemented by multiple devices. The three-dimensional data encoding device may also include some of the multiple processing units included in the three-dimensional data encoding system 101.

The three-dimensional data encoding system 101 includes a three-dimensional data generation system 111, a presentation unit 112, an encoding unit 113, a multiplexing unit 114, an input/output unit 115, and a control unit 116. The three-dimensional data generation system 111 includes a sensor information acquisition unit 117 and a three-dimensional data generation unit 118.

The sensor information acquisition unit 117 acquires sensor information (sensor signals) from the sensor terminal 103 and outputs the sensor information to the three-dimensional data generation unit 118. The three-dimensional data generation unit 118 generates three-dimensional data from the sensor information and outputs the three-dimensional data to the encoding unit 113.

The presentation unit 112 presents the sensor information or three-dimensional data to the user. For example, the presentation unit 112 displays information or an image based on the sensor information or three-dimensional data.

The encoding unit 113 encodes (compresses) the three-dimensional data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 114. The additional information includes, for example, sensor information.

The multiplexing unit 114 generates multiplexed data by multiplexing the coded data input from the coding unit 113, control information, and additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

The input/output unit 115 (e.g., a communication unit or interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as internal memory. The control unit 116 (or application execution unit) controls each processing unit. In other words, the control unit 116 controls encoding, multiplexing, etc.

In addition, the sensor information may be input to the encoding unit 113 or the multiplexing unit 114. Furthermore, the input/output unit 115 may output the three-dimensional data or encoded data directly to the outside.

The transmission signal (multiplexed data) output from the three-dimensional data encoding system 101 is input to the three-dimensional data decoding system 102 via the external connection unit 104.

The three-dimensional data decoding system 102 generates three-dimensional data such as three-dimensional point cloud data or three-dimensional mesh data by decoding the encoded data or multiplexed data. The three-dimensional data decoding system 102 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data decoding device may also include some of the multiple processing units included in the three-dimensional data decoding system 102.

The three-dimensional data decoding system 102 includes a sensor information acquisition unit 121, an input/output unit 122, a demultiplexing unit 123, a decoding unit 124, a presentation unit 125, a user interface 126, and a control unit 127.

The sensor information acquisition unit 121 acquires sensor information (sensor signals) from the sensor terminal 103.

The input/output unit 122 acquires the transmission signal, decodes the multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer unit 123.

The demultiplexing unit 123 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 124.

The decoding unit 124 reconstructs the three-dimensional data by decoding the encoded data.

The presentation unit 125 presents the three-dimensional data to the user. For example, the presentation unit 125 displays information or images based on the three-dimensional data. The user interface 126 acquires instructions based on user operations. The control unit 127 (or application execution unit) controls each processing unit. In other words, the control unit 127 controls demultiplexing, decoding, presentation, etc.

The input/output unit 122 may acquire the three-dimensional data or encoded data directly from outside. The presentation unit 125 may also acquire additional information such as sensor information and present information based on the additional information. The presentation unit 125 may also present information based on user instructions acquired by the user interface 126.

The sensor terminal 103 generates sensor information, which is information obtained by a sensor. The sensor terminal 103 is a terminal equipped with a sensor or camera, and may be, for example, a moving object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

Sensor information that can be acquired by the sensor terminal 103 includes, for example, (1) the distance between the sensor terminal 103 and an object, or the reflectance of the object, obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between the camera and the object, or the reflectance of the object, obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the sensor's attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.

The external connection unit 104 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, etc.

Next, we will explain three-dimensional point cloud data (hereinafter also referred to as point cloud data). Figure 2 is a diagram showing the structure of point cloud data. Figure 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is described.

Point cloud data contains data for multiple points. The data for each point includes location information (three-dimensional coordinates) and attribute information for that location information. A collection of multiple points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

Position information such as three-dimensional coordinates is sometimes called geometry. Furthermore, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information with different attribute types may be associated with one piece of location information. Furthermore, multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

The data file configuration example shown in Figure 3 is an example where there is a one-to-one correspondence between location information and attribute information, and shows the location information and attribute information of N points that make up the point cloud data.

Position information is, for example, information on the three axes x, y, and z. Attribute information is, for example, RGB color information. A typical data file is a ply file.

Next, we will explain three-dimensional mesh data (hereinafter also referred to as mesh data). Figure 4 is a diagram showing an example of the structure of mesh data. Figure 5 is a diagram showing an example of the structure of a data file in which information about mesh data is described.

Mesh data is a data format used in CG (Computer Graphics). Mesh data represents the three-dimensional shape of an object through a collection of surface information. The surface information is polygons such as triangles or quadrangles, and is also called a polygon or polygon mesh.

Mesh data consists of a 3D point cloud (a set of points with 3D position information and attribute information corresponding to that position information), as well as a set of 3D points as vertices, edges connecting two vertices, and faces enclosed by the edges.

Vertices (also expressed as "vertex" or "position") may have attribute information such as color information, reflectance, or normal vectors for the point. Information indicating the relationship between the vertices that make up an edge or face is also called connectivity. The front and back of a face can be expressed by the direction of the normal vector for the point. Mesh data may also have attribute information for faces.

An example of a mesh data file format is an object file. The data file indicates the position information G(1) to G(N) of the N vertices that make up the mesh, and the vertex attribute information A(1) to A(N). Note that the data file does not have to include attribute information. Furthermore, the attribute information does not have to correspond one-to-one to the vertices. Note that Figure 5 shows an example in which a data file has 1 to M pieces of attribute information A2.

Face information is represented by a combination of vertex indices. n[1, 3, 4] indicates that the face in question is a triangular face made up of three vertices, n=1, n=3, and n=4. Furthermore, m[2, 4, 6] indicates that the attribute information for m=1, m=4, and m=6 corresponds to the three vertices, respectively.

Alternatively, attribute information may be recorded in a file separate from the data file, with the data file indicating its pointer information. For example, attribute information may be stored in a two-dimensional attribute map file, with the file name of the attribute map and the two-dimensional coordinates in the attribute map recorded in attribute information A2 of the data file. Either method can be used to specify attribute information for a point.

Next, we will explain the types of three-dimensional data (point cloud data or mesh data). Figure 6 is a diagram showing the types of three-dimensional data. As shown in Figure 6, three-dimensional data includes static objects and dynamic objects.

A static object is three-dimensional data at any time (a certain time). A dynamic object is three-dimensional data that changes over time. Hereinafter, three-dimensional point cloud data at a certain time will be referred to as a PCC (Point Cloud Compression) frame, or frame. Also, three-dimensional mesh data at a certain time will be referred to as a mesh frame, or frame.

The multiple points that make up an object may have certain limitations on the area, pixels, or number of points, as in normal video data. Also, the multiple points that make up an object may not have a limited area, as in map information.

Furthermore, there may be point cloud data or mesh data of various densities, and there may be sparse point cloud data or mesh data and dense point cloud data or mesh data.

The details of each processing unit are explained below. Sensor information is acquired by various methods, such as distance sensors such as LIDAR or range finders, stereo cameras, or a combination of multiple monocular cameras. The three-dimensional data generation unit 118 generates three-dimensional data based on the sensor information acquired by the sensor information acquisition unit 117. The three-dimensional data generation unit 118 generates position information as three-dimensional data and adds attribute information for that position information to the position information.

The three-dimensional data generation unit 118 may process the three-dimensional data when generating position information or adding attribute information. For example, the three-dimensional data generation unit 118 may reduce the amount of data by deleting point clouds with overlapping positions. The three-dimensional data generation unit 118 may also transform the position information (such as by shifting the position, rotating, or normalizing). The three-dimensional data generation unit 118 may also generate mesh data from the point cloud data. The three-dimensional data generation unit 118 may also render the attribute information.

In FIG. 1, the three-dimensional data generation system 111 is included in the three-dimensional data encoding system 101, but it may also be provided independently outside the three-dimensional data encoding system 101.

The encoding unit 113 generates encoded data by encoding the three-dimensional data. There are the following encoding methods. The first is an encoding method that uses position information, and this encoding method will be referred to as the first encoding method hereafter. The second is an encoding method that uses a video codec, and this encoding method will be referred to as the second encoding method hereafter.

The decoding unit 124 decodes the encoded data to decode the point cloud data. The multiplexing unit 114 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 114 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 114 may also multiplex sensor information or attribute information related to the three-dimensional data.

Multiplexing methods or file formats include ISOBMFF, and ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RTP.

The demultiplexing unit 123 extracts encoded data, other media, time information, etc. from the multiplexed data.

The input/output unit 115 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input/output unit 115 may communicate with other devices via the Internet, or with a storage unit such as a cloud server.

Communication protocols include http, ftp, TCP, and UDP. A pull-type communication method or a push-type communication method may be used.

Either wired or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable may be used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or millimeter waves may be used.

Furthermore, broadcasting systems such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 are used.

[First Encoding Method]
Hereinafter, encoding and decoding methods for three-dimensional point clouds or three-dimensional meshes will be described. When the device, process, or syntax in this disclosure relates to encoding or decoding processing of point cloud data, it can also be applied to encoding or decoding of vertices in a three-dimensional mesh. Furthermore, the disclosure regarding encoding or decoding processing of vertices in a three-dimensional mesh can also be applied to encoding or decoding of point cloud data. Furthermore, the disclosure regarding encoding or decoding processing of attribute information of point cloud data may also be applied to encoding or decoding processing of face information, or attribute information for faces or vertices in a three-dimensional mesh. Furthermore, processing may be shared between point cloud encoding and mesh encoding. By sharing processing, it may be possible to reduce the scale of the circuit or software.

FIG. 7 is a diagram showing the configuration of a first encoding unit 130, which is an example of the encoding unit 113 that performs encoding using the first encoding method. FIG. 8 is a block diagram of the first encoding unit 130. The first encoding unit 130 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 130 includes a position information encoding unit 131, an attribute information encoding unit 132, an additional information encoding unit 133, and a multiplexing unit 134.

The first encoding unit 130 is characterized by performing encoding with consideration of three-dimensional structure. The first encoding unit 130 is also characterized by the attribute information encoding unit 132 performing encoding using information obtained from the position information encoding unit 131. The first encoding method is also known as G-PCC (Geometry-based PCC).

The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to the position information encoding unit 131, the attribute information is input to the attribute information encoding unit 132, and the additional information is input to the additional information encoding unit 133.

When mesh data is encoded, the position information of the vertex information is input to the position information encoding unit 131, and the face information, or attribute information for the faces and vertices, is input to the attribute information encoding unit 132.

The position information encoding unit 131 encodes the position information to generate encoded position information (Compressed Geometry), which is encoded data. For example, the position information encoding unit 131 encodes the position information using an N-ary tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. Furthermore, the node that includes a point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined hierarchy or node falls below a threshold.

The attribute information encoding unit 132 generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding using the configuration information generated by the position information encoding unit 131. For example, the attribute information encoding unit 132 determines a reference point (reference node) to reference when encoding the target point (target node) to be processed, based on the octree structure generated by the position information encoding unit 131. For example, the attribute information encoding unit 132 references a peripheral or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

Furthermore, the encoding process for attribute information may include at least one of quantization, prediction, and arithmetic coding. In this case, referencing means using a reference node to calculate a predicted value for attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node contains a point group) to determine encoding parameters. For example, encoding parameters include quantization parameters in quantization, or context in arithmetic coding.

The additional information encoding unit 133 generates encoded additional information (Compressed MetaData), which is encoded data, by encoding compressible data from the additional information.

The multiplexing unit 134 generates a compressed stream, which is encoded data, by multiplexing the encoding position information, encoding attribute information, encoding additional information, and other additional information. The generated compressed stream is output to a processing unit in the system layer (not shown).

Next, we will explain the first decoding unit 140, which is an example of the decoding unit 124 that performs decoding using the first encoding method. Figure 9 is a diagram showing the configuration of the first decoding unit 140. Figure 10 is a block diagram of the first decoding unit 140. The first decoding unit 140 generates point cloud data by decoding, using the first encoding method, coded data (coded stream). This first decoding unit 140 includes a demultiplexing unit 141, a position information decoding unit 142, an attribute information decoding unit 143, and an additional information decoding unit 144.

An encoded stream (compressed stream), which is encoded data, is input to the first decoding unit 140 from a system layer processing unit (not shown).

The demultiplexing unit 141 separates the encoded position information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

The position information decoding unit 142 generates position information by decoding the encoded position information. For example, the position information decoding unit 142 restores the position information of a point group expressed in three-dimensional coordinates from the encoded position information expressed in an N-ary tree structure such as an octree.

The attribute information decoding unit 143 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 142. For example, the attribute information decoding unit 143 determines the reference point (reference node) to reference when decoding the target point (target node) to be processed, based on the octree structure obtained by the position information decoding unit 142. For example, the attribute information decoding unit 143 references a peripheral or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

Furthermore, the attribute information decoding process may include at least one of inverse quantization, prediction, and arithmetic decoding. In this case, referencing means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node contains a point group) to determine decoding parameters. For example, decoding parameters include quantization parameters in inverse quantization, or context in arithmetic decoding.

The additional information decoding unit 144 generates additional information by decoding the encoded additional information. The first decoding unit 140 uses the additional information required for decoding the position information and attribute information during decoding, and outputs the additional information required for the application to the outside.

[Second Encoding Method]
Next, a second encoding unit 150, which is an example of the encoding unit 113 that performs encoding using the second encoding method, will be described. Fig. 11 is a diagram showing the configuration of the second encoding unit 150. Fig. 12 is a block diagram of the second encoding unit 150.

The second encoding unit 150 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. This second encoding unit 150 includes an additional information generation unit 151, a position image generation unit 152, an attribute image generation unit 153, a video encoding unit 154, an additional information encoding unit 155, and a multiplexing unit 156.

The second encoding unit 150 is characterized by generating position images and attribute images by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position images and attribute images using an existing video encoding method. The second encoding method is also called VPCC (Video-based PCC).

The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

The additional information generation unit 151 generates map information for multiple two-dimensional images by projecting three-dimensional structures onto two-dimensional images.

The position image generation unit 152 generates a position image (Geometry Image) based on the position information and the map information generated by the additional information generation unit 151. This position image is, for example, a distance image in which distance (Depth) is indicated as pixel values. Note that this distance image may be an image of multiple point clouds viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

The attribute image generation unit 153 generates an attribute image based on the attribute information and the map information generated by the additional information generation unit 151. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is represented as pixel values. Note that this image may be an image in which multiple point clouds are viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

When mesh data is encoded, the position information of the vertex information is input to the position image generation unit 152, and the face information or attribute information for the faces and vertices is input to the attribute image generation unit 153.

The video encoding unit 154 generates encoded data, an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), by encoding the position image and attribute image using a video encoding method. Any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC.

The additional information encoding unit 155 generates encoded additional information (Compressed MetaData) by encoding the additional information and map information contained in the point cloud data.

The multiplexing unit 156 generates a compressed stream, which is encoded data, by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit in the system layer (not shown).

Next, we will explain the second decoding unit 160, which is an example of the decoding unit 124 that performs decoding using the second encoding method. Figure 13 is a diagram showing the configuration of the second decoding unit 160. Figure 14 is a block diagram of the second decoding unit 160. The second decoding unit 160 generates point cloud data by decoding, using the second encoding method, coded data (coded stream). This second decoding unit 160 includes a demultiplexing unit 161, a video decoding unit 162, an additional information decoding unit 163, a position information generation unit 164, and an attribute information generation unit 165.

The encoded stream (compressed stream), which is encoded data, is input to the second decoding unit 160 from a system layer processing unit (not shown).

The demultiplexing unit 161 separates the encoded position image (Compressed Geometry Image), encoded attribute image (Compressed Attribute Image), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

The video decoding unit 162 generates a position image and an attribute image by decoding the encoded position image and encoded attribute image using a video encoding method. Any known encoding method may be used as the video encoding method. For example, the video encoding method may be AVC or HEVC.

The additional information decoding unit 163 decodes the encoded additional information to generate additional information including map information, etc.

The location information generation unit 164 generates location information using the location image and map information. The attribute information generation unit 165 generates attribute information using the attribute image and map information.

The second decoding unit 160 uses the additional information required for decoding during decoding and outputs the additional information required for the application to the outside.

[Position information encoding in the first encoding method]
15 is a block diagram showing an example configuration of the position information encoding unit 131. The position information encoding unit 131 includes an octree encoding unit 171 and a predictive tree encoding unit 172. The octree encoding unit 171 generates encoded position information and metadata by encoding the position information using an encoding method that uses an octree (octree encoding). The predictive tree encoding unit 172 generates encoded position information and metadata by encoding the position information using an encoding method that uses a predictive tree (predictive tree encoding).

The position information encoding unit 131 encodes the position information using either octree encoding or predictive tree encoding, or both. The position information encoding unit 131 may switch between these two encoding methods, may use an encoding method other than these two methods, or may include the position information in the bitstream as raw data without encoding it. Information indicating the encoding method of the encoded data is stored in metadata and notified to the decoding device.

FIG. 16 is a block diagram showing an example configuration of the position information decoding unit 142. The position information decoding unit 142 includes an octree decoding unit 173 and a predictive tree decoding unit 174. The octree decoding unit 173 generates position information by decoding the encoded position information using a decoding method that uses an octree (octree decoding). The predictive tree decoding unit 174 generates position information by decoding the encoded position information using a decoding method that uses a predictive tree (predictive tree decoding). The position information decoding unit 142 also performs decoding using the encoding method notified by the metadata.

Next, an example configuration of the position information encoding unit will be described. Figure 17 is a block diagram of the octree encoding unit 171 according to this embodiment. The octree encoding unit 171 includes an octree generation unit 181, a geometric information calculation unit 182, a coding table selection unit 183, and an entropy encoding unit 184.

The octree generation unit 181 generates, for example, an occupancy tree from the input position information, and generates an occupancy code for each node in the occupancy tree. The geometric information calculation unit 182 obtains information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 182 calculates the occupancy information of the adjacent node (information indicating whether the adjacent node is an occupied node) from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 182 may also store encoded nodes in a list and search for adjacent nodes from within that list. The geometric information calculation unit 182 may also switch adjacent nodes depending on the position of the target node within the parent node.

The coding table selection unit 183 selects the coding table to be used for entropy coding of the target node using the occupancy information of adjacent nodes calculated by the geometric information calculation unit 182. For example, the coding table selection unit 183 may generate a bit string using the occupancy information of adjacent nodes, and select a coding table for the index number generated from that bit string.

The entropy coding unit 184 generates coding position information and metadata by entropy coding the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 184 may add information indicating the selected coding table to the coding position information.

The octree representation and the scanning order of position information are explained below. Position information (position data) is converted (octreeized) into an octree structure and then encoded. The octree structure consists of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Figure 18 is a diagram showing an example of the structure of position information containing multiple voxels. Figure 19 is a diagram showing an example of the position information shown in Figure 18 converted into an octree structure. Here, of the leaves shown in Figure 19, leaves 1, 2, and 3 represent voxels VXL1, VXL2, and VXL3 shown in Figure 18, respectively, and represent a VXL containing a point cloud (hereinafter referred to as a valid VXL).

Specifically, node 1 corresponds to the entire space that contains the position information in Figure 18. The entire space corresponding to node 1 is divided into eight nodes, and of these eight nodes, the node that contains a valid VXL is further divided into eight nodes or leaves, and this process is repeated for each level of the tree structure. Here, each node corresponds to a subspace, and node information contains information (occupancy code) that indicates the position of the next node or leaf after division. In addition, the lowest-level block is set as a leaf, and leaf information such as the number of points contained in the leaf is stored.

Next, an example configuration of the position information decoding unit will be described. Figure 20 is a block diagram of the octree decoding unit 173 according to this embodiment. The octree decoding unit 173 includes an octree generation unit 191, a geometric information calculation unit 192, an encoding table selection unit 193, and an entropy decoding unit 194.

The octree generator 191 generates an octree for a space (node) using header information or metadata from the bitstream. For example, the octree generator 191 generates a large space (root node) using the dimensions of the x-axis, y-axis, and z-axis of a space added to the header information, and then generates an octree by dividing that space into two along the x-axis, y-axis, and z-axis to generate eight small spaces A (nodes A0 to A7). Nodes A0 to A7 are set in order as target nodes.

The geometric information calculation unit 192 obtains occupancy information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 192 calculates the occupancy information of the adjacent node from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 192 may also store decoded nodes in a list and search for adjacent nodes from within that list. The geometric information calculation unit 192 may also switch adjacent nodes depending on the position of the target node within the parent node.

The coding table selection unit 193 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of adjacent nodes calculated by the geometric information calculation unit 192. For example, the coding table selection unit 193 may generate a bit string using the occupancy information of adjacent nodes, and select a coding table for the index number generated from that bit string.

The entropy decoding unit 194 generates position information by entropy decoding the occupancy code of the target node using the selected encoding table. The entropy decoding unit 194 may also obtain information about the selected encoding table by decoding it from the bitstream, and entropy decode the occupancy code of the target node using the encoding table indicated by that information.

[Attribute information encoding in the first encoding method]
The configurations of the attribute information encoder and the attribute information decoder will be described below. Fig. 21 is a block diagram showing an example configuration of the attribute information encoder 132. The attribute information encoder may include multiple encoders that execute different encoding methods. For example, the attribute information encoder may switch between the following two methods depending on the use case:

The attribute information encoding unit 132 includes an LoD attribute information encoding unit 201 and a transformed attribute information encoding unit 202. The LoD attribute information encoding unit 201 uses the position information of each 3D point to classify the 3D points into multiple layers, predicts the attribute information of the 3D points belonging to each layer, and encodes the prediction residual. Here, each classified layer is called an LoD (Level of Detail).

The transformed attribute information encoding unit 202 encodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information encoding unit 202 applies RAHT or Haar transform to each piece of attribute information based on the position information of the three-dimensional points, generating high-frequency and low-frequency components for each layer, and then encodes these values using quantization, entropy coding, etc.

FIG. 22 is a block diagram showing an example configuration of the attribute information decoding unit 143. The attribute information decoding unit may include multiple decoding units that execute different decoding methods. For example, the attribute information decoding unit may switch between the following two methods based on information contained in the header or metadata.

The attribute information decoding unit 143 includes an LoD attribute information decoding unit 203 and a converted attribute information decoding unit 204. The LoD attribute information decoding unit 203 classifies each 3D point into multiple layers using the position information of the 3D points, and decodes the attribute values while predicting the attribute information of the 3D points belonging to each layer.

The transformed attribute information decoding unit 204 decodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information decoding unit 204 decodes the attribute values by applying an inverse RAHT or inverse Haar transform to the high-frequency and low-frequency components of each attribute value based on the position information of the three-dimensional points.

FIG. 23 is a block diagram of a transformed attribute information encoding unit 202, which is an example of the transformed attribute information encoding unit 202. The transformed attribute information encoding unit 202 includes a sorting unit 211, a Haar transform unit 212, a quantization unit 213, an inverse quantization unit 214, an inverse Haar transform unit 215, a memory 216, and an arithmetic encoding unit 217.

The sorting unit 211 generates a Morton code using the position information of the 3D points and sorts the multiple 3D points in Morton code order. The Haar transform unit 212 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 213 quantizes the coding coefficients of the attribute information.

The inverse quantization unit 214 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 215 applies an inverse Haar transform to the coding coefficients. The memory 216 stores values of attribute information for multiple decoded 3D points. For example, the attribute information for decoded 3D points stored in the memory 216 may be used for predicting uncoded 3D points.

The arithmetic coding unit 217 calculates ZeroCnt from the coding coefficients after quantization and arithmetically codes ZeroCnt. The arithmetic coding unit 217 also arithmetically codes the non-zero coding coefficients after quantization. The arithmetic coding unit 217 may also binarize the coding coefficients before arithmetically coding them. The arithmetic coding unit 217 may also generate and code various header information.

FIG. 24 is a block diagram of a transformed attribute information decoding unit 204, which is an example of the transformed attribute information decoding unit 204. The transformed attribute information decoding unit 204 includes an arithmetic decoding unit 221, an inverse quantization unit 222, an inverse Haar transform unit 223, and a memory 224.

The arithmetic decoding unit 221 arithmetically decodes the ZeroCnt and coding coefficients contained in the bitstream. The arithmetic decoding unit 221 may also decode various header information.

The inverse quantization unit 222 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 223 applies an inverse Haar transform to the coding coefficients after inverse quantization. The memory 224 stores values of attribute information for multiple decoded 3D points. For example, the attribute information for decoded 3D points stored in the memory 224 may be used to predict undecoded 3D points.

[Slice division]
The encoding device may divide the three-dimensional data into one or more pieces of three-dimensional data and encode them. The divided three-dimensional point cloud is called a slice. A slice is a set of points each having position information (geometry) and attribute information (attribute).

FIG. 25 is a block diagram showing an example configuration of a three-dimensional data encoding device in this case. The three-dimensional data encoding device includes a data division unit 231 and an encoding unit 232.

The data division unit 231 divides the three-dimensional data to generate multiple pieces of divided three-dimensional data. Each piece of divided three-dimensional data corresponds to a slice. The encoding unit 232 generates encoded data by encoding the position information and attribute information of each piece of divided three-dimensional data (slice).

FIG. 26 is a block diagram showing an example configuration of a three-dimensional data decoding device in this case. The three-dimensional data decoding device includes a decoding unit 233 and a data combining unit 234. The decoding unit 233 generates multiple pieces of divided three-dimensional data by decoding the encoded data. Each piece of divided three-dimensional data includes slice position information and attribute information. The data combining unit 234 restores the three-dimensional data by combining the multiple pieces of divided three-dimensional data.

Note that during encoding, there may or may not be a dependency between slices. If there is no dependency, slices can be encoded or decoded independently. Therefore, processing time can be reduced by processing multiple slices in parallel. Furthermore, the amount of processing can be reduced by partial decoding, which decodes only some of the slices.

If there is a dependency, an identifier indicating the dependency is stored in the bitstream, and encoding or decoding is performed starting with the dependent data (referenced data).

The number of divisions or division method may be any method. The three-dimensional data encoding device may determine the shape of the object and divide the point cloud for each object, or may perform division based on the number of points contained in a slice. The three-dimensional data encoding device may also use map information or location information to perform division based on whether the point cloud is contained in the three-dimensional space (tile information).

Figure 27 is a diagram showing the relationship between tiles and slices. As shown in Figure 27, tiles correspond to three-dimensional space, and slices correspond to divided three-dimensional point clouds. Note that multiple tiles may overlap. This enables parallel processing in adaptive encoding or decoding according to content or objects, improving the flexibility of point cloud encoding systems or point cloud decoding systems.

[Structure of encoded data]
The position information and attribute information for each slice are coded, and at least a portion of the coded data is stored in the payload of a data unit. A header is also added to the payload.

If there are multiple pieces of attribute information for each point, each piece of attribute information is stored in a data unit. Figure 28 is a diagram showing an example of the configuration of a bit stream (encoded data). Figure 28 shows the encoded data of two frames of point clouds with two types of attribute information. The bit stream has a geometry (position information) data unit (Geometry Data Unit: Geom) and two attribute (attribute information) data units (Attribute Data Unit: Attr(0), Attr(1)) for each slice. For example, if each point has two pieces of attribute information, color and reflectance, the encoded data for color is stored in Attr(0) and the encoded data for reflectance is stored in Attr(1). The header of the geometry data unit stores the slice identifier (slice_id). The header of the attribute data unit indicates the slice_id of the corresponding (referenced) geometry data unit. Note that a data unit is sometimes called a slice, and a data unit header is sometimes called a slice header.

In addition, metadata related to encoding location information is stored in the Geometry Parameter Set (GPS). Metadata related to encoding attribute information is stored in the Attribute Parameter Set (APS). Metadata common to multiple PCC frames (PCC sequences) is stored in the Sequence Parameter Set (SPS).

Each data unit and parameter set is converted into NAL (Network Access Layer) unit or TLV (Type Length Value) unit format, and a data sequence (stream) of the units is output.

Figure 29 is a block diagram showing an example configuration of a three-dimensional data encoding device that outputs a stream of TLV units. The three-dimensional data encoding device includes an encoding unit 241 and a TLV storage unit 242. The encoding unit 241 generates encoded data by encoding point cloud data. The TLV storage unit 242 generates a TLV stream (a single stream of TLV units) by storing (encapsulating) the encoded data in multiple TLV units.

A unit includes a Type that indicates the data type, a Length that indicates length information, and a Value that stores the data indicated by the Type. Note that a unit may also be in a different format, such as one that does not include length information.

Figure 28 shows a sequence of units that store data units and parameter sets. The arrows in Figure 28 indicate dependencies related to the decoding of encoded data. The source of the arrow depends on the data at the end of the arrow, and the header of the original data indicates the identifier of the data at the end of the arrow (reference destination). The three-dimensional data decoding device decodes the data at the end of the arrow and uses that decoded data to decode the data at the source of the arrow. For example, a geometry data unit (Geom) indicates the GPS ID and SPS ID corresponding to that geometry data unit. An attribute data unit (Attr) indicates the APS ID corresponding to that attribute data unit.

One GPS and one APS may be provided for each frame. Alternatively, if the encoding method is changed for each slice, a GPS and an APS may be provided for each slice. Also, a GPS and an APS may be shared by multiple frames (sequences).

APS may also be shared by multiple attribute information. The referenced parameter set is sent before the referenced parameter set.

When inter-frame prediction is used, or when there is a dependency between data units of frames, a GOF (Group of Frame) containing multiple frames may be configured. A GOF is a random access unit, and the first slice of a GOF has no dependency and is the slice from which decoding begins. The stream may have a delimiter indicating the frame boundary or GOF boundary, or a TLV unit indicating the boundary.

Next, we will explain the structure of encoded data and how it is stored in NAL units.

For example, a data format is defined for each type of encoded data. Figure 30 shows examples of encoded data and NAL units.

For example, as shown in FIG. 30, the encoded data includes a header and a payload. The encoded data may also include length information indicating the length (amount of data) of the encoded data, header, or payload. The encoded data may also be in a TLV format that includes length information. The encoded data does not necessarily have to include a header.

The header includes, for example, identification information for identifying the data. This identification information indicates, for example, the data type or frame number.

The header includes, for example, identification information indicating a reference relationship. This identification information is stored in the header when, for example, there is a dependency between data, and is information used by the referencing source to reference the referenced data. For example, the header of the referenced data includes identification information for identifying the data in question. The header of the referencing source includes identification information indicating the referenced data.

In addition, if the reference destination or reference source can be identified or derived from other information, the identification information for specifying the data or the identification information indicating the reference relationship may be omitted.

The three-dimensional data encoding device stores encoded data in the payload of a NAL unit. The NAL unit header contains pcc_nal_unit_type, which is identification information for the encoded data. Figure 31 shows an example of the semantics of pcc_nal_unit_type.

As shown in Figure 31, when pcc_codec_type is Codec 1 (Codec1: first encoding method), the values 0 to 10 of pcc_nal_unit_type represent the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), and position PS (Geom.PS) in Codec 1. , attribute XPS (AttrX.PS), attribute YPS (AttrY.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Header). Values 11 and above are assigned as spares for codec 1.

Below, a modified example of this embodiment will be described. PS has levels, such as frame-level PS, sequence-level PS, and PCC sequence-level PS. If the PCC sequence level is the higher level and the frame level is the lower level, the following method may be used to store parameters.

The default PS value is indicated in the higher PS. Also, if the value of the lower PS differs from the value of the higher PS, the PS value is indicated in the lower PS. Alternatively, the PS value is not written in the higher PS, but written in the lower PS. Alternatively, information on whether the PS value is to be written in the lower PS, the higher PS, or both is written in either the lower PS or the higher PS, or both. Alternatively, the lower PS may be merged with the higher PS. Alternatively, if the lower PS and the higher PS overlap, the three-dimensional data encoding device may omit sending one of them.

The three-dimensional data encoding device may also divide the data into slices or tiles and transmit the divided data. The divided data includes information for identifying the divided data, and the parameters used to decode the divided data are included in the parameter set. In this case, pcc_nal_unit_type defines an identifier indicating that the data stores data or parameters related to tiles or slices.

[Gaussian data]
Gaussian data (Gaussian Splatting 3D data) will be described below. First, the configuration of a system for encoding or decoding three-dimensional data generated using the Gaussian Splatting method will be described.

FIG. 32 is a diagram showing part of the processing in the three-dimensional data generation system shown in FIG. 1. The sensor information input unit 301 and three-dimensional data generation unit 302 shown in FIG. 32 correspond to, for example, the sensor information acquisition unit 117 and three-dimensional data generation unit 118 shown in FIG. 1.

The sensor information input unit 301 acquires point cloud data (three-dimensional point cloud data) and outputs the acquired point cloud data to the three-dimensional data generation unit 302. The three-dimensional data generation unit 302 generates Gaussian data (Gaussian splatting data) from the point cloud data and outputs the generated Gaussian data.

Specifically, the three-dimensional data generation unit 302 first generates a grid according to the density of the input point cloud data. Each point of the point cloud data is projected onto a nearby grid. At this time, attribute information associated with the point, such as color, reflectance, or normal, is also linked to the grid at the same time. The three-dimensional data generation unit 302 derives parameters such as the mean value and variance of the Gaussian function from the information projected onto the grid.

Note that the method for generating Gaussian data is not limited to the above. For example, Gaussian data may be generated by machine learning using point cloud data generated from multiple two-dimensional images using methods such as Structure from Motion (SfM).

Gaussian data is a collection of multiple 3D Gaussians (ellipsoids). Each 3D Gaussian contains data such as three-dimensional point coordinates, a 3x3 covariance matrix, color, and transparency. The above 3D Gaussian data can also be converted into components described in the PLY format and expressed.

Figure 33 is a diagram showing the components of Gaussian data written in PLY format. The Gaussian data shown in Figure 33 includes position, rotation, scale, SH coefficient, and transparency. Note that transparency is also called transparency, transparency, or opacity.

Position is the three-dimensional coordinate of a three-dimensional point. Scale and Orientation are parameters of the three-dimensional covariance matrix. Also, SH coefficients are the coefficients of spherical harmonics when color information is expressed as spherical harmonics.

In other words, the Gaussian data includes the reference three-dimensional coordinates (position) for each 3D Gaussian, and information associated with the three-dimensional coordinates (rotation, scale, SH coefficient, transmittance).

Note that the configuration of Gaussian data is not limited to the above. For example, Gaussian data does not have to include a covariance matrix. For example, Gaussian data includes a position (Position) indicating the three-dimensional coordinates of a three-dimensional point, a scale (Scale) indicating the scale along each axis, a color (Color) indicating, for example, RGB, and a transparency (Transparency).

In addition, instead of multiple ellipsoids, the Gaussian data may include multiple cubes (voxels) that divide 3D space. Each voxel has attribute information. For example, each voxel may include a position (Position) indicating the three-dimensional coordinates of the voxel's center point, attribute information such as color, reflectance, or material, and a scale (Scale) indicating the size of the voxel.

Furthermore, the position (Position) may be expressed as two-dimensional point coordinates (x, y) rather than three-dimensional coordinates. In this case, the scale (Scale) may indicate the size or shape of the ellipsoid on a two-dimensional plane. Specifically, the Gaussian data may include a position (Position) indicating a position on a two-dimensional plane, a scale vector (Scale Vector) indicating the scale (Su, Sv) along each axis, and rotation information (Rotation) indicating the orientation of the ellipsoid.

Note that the format for writing Gaussian data is not limited to the PLY format. For example, the format for writing Gaussian data may be the SPZ format.

[Gaussian data rendering]
Next, the rendering process of Gaussian data will be described. FIG. 34 is a diagram showing the rendering process of Gaussian data. As shown in FIG. 34, the rendering unit 303 generates a 3D model (three-dimensional model) by rendering the Gaussian data. This makes it possible to present the generated 3D model in an application such as a 3D display device. The 3D display device is, for example, a device that displays a stereoscopic image in space like a hologram, or a naked-eye 3D display device.

Figure 35 shows an example of the number of SH coefficients in a spherical harmonic function. The number of elements (number of SH coefficients) in a spherical harmonic function is determined according to the level indicating the resolution. For example, the higher the level (the larger the level number), the higher the resolution. For example, the higher the level, the more high-frequency component information is added.

For example, when expressing level 2, the spherical harmonic function has a total of 9 elements (SH coefficients) from levels 0 to 2. For example, when expressing level 3, the spherical harmonic function has a total of 16 elements (SH coefficients) from levels 0 to 3.

For example, the example shown in Figure 33 shows a spherical harmonic function for a level 3 color, where each of the three color elements (R, G, B) is expressed by 16 SH coefficients. Therefore, the spherical harmonic function has 16 x 3 = 48 elements. It is also possible to define levels 4 and above.

By expressing color using spherical harmonic functions, it is possible to express three-dimensional color information. Figure 36 is a diagram showing an example of the input and output of spherical harmonic functions. For example, as shown in Figure 36, by inputting viewpoint information into the color spherical harmonic function 304, color information (R, G, B) seen from that viewpoint is output.

Figure 37 is a diagram showing the rendering process. As shown in Figure 37, the rendering unit 305 generates a 3D model or 2D image viewed from a specified viewpoint based on Gaussian data and the specified viewpoint. The generated 3D model can be presented in an application such as a VR display. The generated 2D image can also be presented in an application such as a 2D image display device.

In addition to color, other attribute information (such as reflectance or infrared information) may also be expressed using spherical harmonic functions. In that case, the Gaussian data includes the SH coefficients of the spherical harmonic functions for each attribute information. Furthermore, the level may be changed depending on the attribute information or resolution.

Gaussian Data Encoding and Decoding
Next, encoding and decoding processes for Gaussian data will be described. FIG. 38 is a block diagram of an encoding device (encoding system) according to this embodiment. This encoding device encodes and multiplexes Gaussian data. FIG. 39 is a block diagram of a decoding device (decoding system) according to this embodiment. This decoding device decodes the Gaussian data and presents a 3D model or a 2D image in an application. The encoding device and decoding device shown in FIGS. 38 and 39 are included in, for example, the three-dimensional data generation system shown in FIG. 1.

The encoding device shown in FIG. 38 includes an encoding unit 311 and a multiplexing unit 312. The encoding unit 311 generates encoded data (bit stream) by encoding the input Gaussian data using a predetermined encoding method, and outputs the generated encoded data to the multiplexing unit 312.

The multiplexing unit 312 generates multiplexed data by multiplexing the input coded data using a predetermined multiplexing method, and outputs the generated multiplexed data. This multiplexed data is stored or transmitted.

The decoding device shown in FIG. 39 includes a demultiplexing unit 321, a decoding unit 322, and an application unit 323. The demultiplexing unit 321 generates coded data by demultiplexing the input multiplexed data using a predetermined multiplexing method, and outputs the generated coded data to the decoding unit 322.

The decoding unit 322 generates Gaussian data by decoding the input coded data using a predetermined coding method (decoding method), and outputs the generated Gaussian data to the application unit 323.

The application unit 323 includes an input interface unit 324, a rendering unit 325, and a presentation unit 326. The input interface unit 324 acquires user operations. For example, the input interface unit 324 generates viewpoint information indicating the viewpoint entered by the user based on the user operations.

The rendering unit 325 generates a three-dimensional model or a two-dimensional image by rendering the input Gaussian data. For example, the rendering unit 325 generates a 3D model or a 2D image viewed from a viewpoint input by the user, which is indicated by the viewpoint information. Note that the rendering unit 325 may also generate a 3D model or a 2D image viewed from a predetermined viewpoint.

The presentation unit 326 presents (displays) the three-dimensional model or two-dimensional image generated by the rendering unit 325. Whether to present a 3D model or a 2D image may be determined, for example, by the following method. For example, the rendering unit 325 first generates a three-dimensional model and generates point coordinate information, as in the process shown in FIG. 34. After viewpoint information is input, the rendering unit 325 generates a 2D image viewed from a specified viewpoint. At that time, the rendering unit 325 may accept information specifying an area along with the specified viewpoint. In this way, by first generating a 3D model and then generating a 2D image of the specified area, the amount of processing can be reduced compared to generating 2D images of the entire area. This allows for faster rendering. Alternatively, whether to present a 3D display (3D model) or a 2D image may be determined depending on the processing power of the device performing the rendering. For example, in the case of a device with low processing power, the device may present a 3D model without generating a 2D image.

(First Aspect)
Encoding Gaussian Data Using G-PCC
40 is a block diagram showing an example of the configuration of an encoding device 330 (three-dimensional data encoding device) according to this embodiment. The encoding device 330 generates a bit stream (encoded data) by encoding the first Gaussian data.

The encoding device 330 includes a first pre-processing unit 331, a second pre-processing unit 332, and a G-PCC encoding unit 333.

The first Gaussian data includes, for example, a covariance matrix. The first pre-processing unit 331 converts the first Gaussian data into second Gaussian data composed of components in a ply format. For example, the components of a ply file include three-dimensional coordinates, scale, rotation, SH coefficients of spherical harmonics, and transmittance, as shown in FIG. 33.

Note that, although an example is described here in which first Gaussian data is input to the encoding device 330, and the encoding device 330 converts the first Gaussian data into second Gaussian data, second Gaussian data may also be input to the encoding device 330. In this case, the encoding device 330 does not need to include the first pre-processing unit 331.

The second pre-processing unit 332 converts the multiple components contained in the second Gaussian data into a format that can be encoded using an encoding method corresponding to each component, and outputs the converted multiple components to the G-PCC encoding unit 333.

The G-PCC encoding unit 333 encodes three-dimensional coordinates using a geometry (position information) encoding method (such as octree encoding, predictive tree encoding, or TriSoup) in the G-PCC encoding method (Geometry-based PCC). The G-PCC encoding unit 333 encodes information associated with three-dimensional coordinates, such as rotation, scale, SH coefficients, and transparency, using an attribute encoding method in the G-PCC encoding method (such as LoD-based encoding (LoD attribute information encoding) or Transform-based encoding (transform attribute information encoding)).

The G-PCC encoding unit 333 includes a position information encoding unit 341, a rotation encoding unit 342, a scale encoding unit 343, an SH coefficient encoding unit 344, a transmittance encoding unit 345, a metadata encoding unit 346, and a multiplexing unit 347.

The position information encoding unit 341 encodes three-dimensional coordinates. The three-dimensional coordinates are reference or representative three-dimensional coordinates for the Gaussian data. For example, the three-dimensional coordinates may be the center coordinates of an ellipse that constitutes the Gaussian data, or may be other coordinates. For example, the three-dimensional coordinates may be the origin coordinates of a grid used to generate the Gaussian data.

Furthermore, the three-dimensional coordinates are not processed by the first pre-processing unit 331, but are input directly to the second pre-processing unit 332. The second pre-processing unit 332 converts the input three-dimensional coordinates into positive integer information (three-dimensional coordinates). The scale and offset values used during the conversion are stored in metadata such as SPS and notified to the decoding device.

The location information encoding unit 341 generates encoded data by encoding the three-dimensional coordinates output from the second pre-processing unit 332 using the G-PCC geometry encoding method. This encoded data is stored in a GDU (Geometry Data Unit) and output. In addition, the metadata required for decoding the GDU is stored in a GPS (Geometry Parameter Set) and output.

Furthermore, if the multiple three-dimensional coordinates of the multiple Gaussian data are sparse (density is less than a predetermined threshold), the position information encoding unit 341 may use a predictive tree encoding method suitable for encoding sparse three-dimensional points. Furthermore, if the multiple three-dimensional coordinates of the multiple Gaussian data are dense (density is equal to or greater than a predetermined threshold), the position information encoding unit 341 may use an octree encoding method suitable for encoding dense three-dimensional points.

The first pre-processing unit 331 converts the covariance matrix contained in the first Gaussian data into rotation values and scale values. The first pre-processing unit 331 converts the color information contained in the first Gaussian data into spherical harmonic functions and outputs SH coefficients in the spherical harmonic functions.

The second pre-processing unit 332 converts the data type of each piece of data contained in the second Gaussian data. Specifically, the second pre-processing unit 332 converts each piece of data into positive integer data so that it can be encoded using the G-PCC encoding method. For example, if the data is float type, the second pre-processing unit 332 converts the data into positive integer data by scaling and offsetting the data. At this time, the scale value and offset value for each piece of attribute information used in the conversion are stored in metadata such as the SPS and notified to the decoding device.

The second pre-processing unit 332 also performs format conversion for each piece of data and mapping processing to map each piece of data to attribute components. For example, if the maximum number of elements (number of dimensions) per attribute component that can be encoded using the G-PCC attribute encoding method is three, the second pre-processing unit 332 converts the input elements into attribute components with three dimensions (three elements) per attribute component. For example, since scale has three elements, it is output as a single attribute component. Furthermore, since rotation has four elements, it is converted into two two-dimensional attribute components. The number of elements for SH coefficients changes for each level. For example, SH coefficients with levels greater than three are converted into multiple attribute components.

At this time, the second pre-processing unit 332 generates mapping information as metadata for each piece of attribute information, indicating to which attribute component each element is mapped.

The rotation encoding unit 342, scale encoding unit 343, SH coefficient encoding unit 344, and transparency encoding unit 345 generate multiple pieces of coded data by encoding the rotation, scale, SH coefficient, and transparency, respectively, using the attribute encoding method in the G-PCC coding method. Each of the multiple pieces of coded data generated is stored in an ADU (Attribute Data Unit) and output. The metadata required for decoding the ADU is stored in an APS (Attribute Parameter Set) and output. In addition, the attribute information is encoded using three-dimensional coordinates.

The metadata encoding unit 346 stores transformation information, including the transformation parameters used in the first pre-processing unit 331 or the second pre-processing unit 332, in metadata such as SPS or SEI. The metadata encoding unit 346 may encode the transformation information and then store it in the metadata. The metadata may also include configuration information for the Gaussian data, or information indicating the correspondence between how the Gaussian data is transmitted using G-PCC encoding.

The multiplexing unit 347 stores the coded data (ADU and GDU) and parameter sets (APS, GPS, and SPS) in TLV units and transmits them as a bit stream (coded data). The bit stream may be multiplexed using a predetermined multiplexing method. The bit stream may also be formatted.

[Gaussian Data Decoding Using G-PCC]
Fig. 41 is a block diagram showing an example configuration of a decoding device 350 (three-dimensional data decoding device) according to this embodiment. The decoding device 350 generates first Gaussian data or second Gaussian data by decoding a bit stream (encoded data). For example, the decoding device 350 decodes the bit stream generated by the encoding device 330 shown in Fig. 40.

The decoding device 350 includes a G-PCC decoding unit 351, a second post-processing unit 352, and a first post-processing unit 353.

The G-PCC decoding unit 351 generates three-dimensional coordinates, rotation, scale, SH coefficients, and transmittance by decoding the bitstream using the G-PCC encoding method. The G-PCC decoding unit 351 includes a demultiplexing unit 361, a position information decoding unit 362, a rotation decoding unit 363, a scale decoding unit 364, an SH coefficient decoding unit 365, a transmittance decoding unit 366, and a metadata decoding unit 367.

The demultiplexing unit 361 analyzes multiple TLV units contained in the input bitstream and generates data units of encoded data such as GDU, ADU, SPS, GPS, and APS. The position information decoding unit 362, rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, transmittance decoding unit 366, and metadata decoding unit 367 use the parameter set information to decode these data units using the respective encoding methods for each attribute component.

The position information decoding unit 362 decodes three-dimensional coordinates from the GDU. The rotation decoding unit 363, scale decoding unit 364, SH coefficient decoding unit 365, and transmittance decoding unit 366 decode attribute information (rotation, scale, SH coefficient, and transmittance) from the ADU. The attribute information is decoded using the three-dimensional coordinates. The metadata decoding unit 367 acquires transformation information, etc. from metadata such as SPS or SEI. The metadata decoding unit 367 may also decode the transformation information from the metadata. The metadata may also include configuration information of the Gaussian data, or information indicating the correspondence of how the Gaussian data is transmitted using G-PCC encoding.

The second post-processing unit 352 generates second Gaussian data by inversely transforming the decoded data. Here, the inverse transformation is the reverse of the transformation process performed by the second pre-processing unit 332, and is performed based on the transformation information (transformation parameters) included in the metadata.

The first post-processing unit 353 generates first Gaussian data by inversely transforming the second Gaussian data. Here, the inverse transform is the reverse of the transformation process performed by the first pre-processing unit 331. In addition, at least one of the first Gaussian data and the second Gaussian data is output. Note that the decoding device 350 does not necessarily have to include the first post-processing unit 353.

[Mapping process]
The second pre-processing unit 332 maps the Gaussian data to the attribute components of the G-PCC. The mapping method and the mapping information generated at that time will be described below.

Below, an example of mapping rotation, scale, and SH coefficients is shown, and an explanation of transparency is omitted. Note that similar processing can also be applied when other attribute information is added.

Figure 42 shows an example of mapping multiple attribute components contained in Gaussian data to attribute components of G-PCC, which is a component of the encoding method. Here, an example is shown in which the maximum number of dimensions (number of subcomponents) that can be encoded when encoding attribute components in G-PCC is 3.

In the following, the attribute components of Gaussian data will be referred to as Gaussian components, and the attribute components of G-PCC will be referred to as G-PCC components. Attribute components will also be referred to simply as components.

For example, the Gaussian component of scale has subcomponents (B1 to B3) with a dimensionality of 3, and is mapped to one G-PCC component with three-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian component of scale corresponds to the identifier of the G-PCC component.

Furthermore, the Gaussian rotation component has subcomponents (A1 to A4) with a dimensionality of four, and cannot be mapped to a single G-PCC component with three-dimensional subcomponents. Therefore, the four rotation subcomponents (A1 to A4) are mapped to two G-PCC components with two-dimensional subcomponents. In this case, the mapping information indicates that the identifier of the Gaussian rotation component and the identifier indicating the subcomponent number included in the attribute component correspond to the identifier of the G-PCC component and the identifier indicating the subcomponent number included in the G-PCC component.

Since there are 16 SH coefficients per level for each three-dimensional (RGB) element, the SH coefficients have a total of 48 dimensional elements (C1 to C48). These 48 dimensional elements are mapped to 16 G-PCC components with three-dimensional subcomponents.

Note that "ID" in Figure 42 is the attribute component ID, which indicates the identification number used to identify the attribute component of the encoded data. "ND" is the number of dimensions, which indicates the number of elements (number of subcomponents) per attribute component.

Furthermore, the division method and mapping method for Gaussian data are not limited to the above examples, and any combination of mapping method and number of dimensions may be used.

In the mapping process, the second pre-processing unit 332 generates mapping information indicating the G-PCC components corresponding to the Gaussian components. This mapping information is stored as metadata in the header or SEI.

The decoding device decodes the mapping information from the bitstream. The second post-processing unit 352 reconstructs the Gaussian data by remapping the decoded data for each G-PCC component into Gaussian data based on the mapping information.

Figure 43 is a diagram showing an example of mapping information. In Figure 43, the mapping information is stored in SEI (Gaussian data attribute mapping information SEI). The mapping information includes the number of Gaussian data components. The number of Gaussian data components indicates the number of attribute information included in the Gaussian data. For example, in the example shown in Figure 42, rotation, scale, and SH coefficients are present, and the number of Gaussian data components is 3.

The mapping information also includes, for each Gaussian component, a Gaussian data component ID, a Gaussian data type, and the number of Gaussian data dimensions.

The Gaussian component ID is identification information for uniquely identifying a Gaussian component. The Gaussian data type indicates the type of Gaussian component. For example, a value of 0 indicates "rotation", a value of 1 indicates "scale", a value of 2 indicates "SH coefficient", and a value of 3 indicates "transmittance". Note that the combinations of value and type here are just examples, and the combinations are not limited to these.

The number of Gaussian data dimensions indicates the number of dimensions of the Gaussian component (number of subcomponents). For example, in the example shown in Figure 42, the number of Gaussian data dimensions for rotation is 4, the number of Gaussian data dimensions for scale is 3, and the number of Gaussian data dimensions for SH coefficients is 48.

The mapping information also includes a G-PCC component ID, a G-PCC dimension ID, and conversion information as information for each dimension of the Gaussian component. The G-PCC subcomponent ID corresponds to the "ID" shown in Figure 42. The G-PCC dimension ID is an identifier that indicates the subcomponent number. For example, C1 included in the SH coefficients shown in Figure 42 has a G-PCC dimension ID of 0, C2 has a G-PCC dimension ID of 1, and C3 has a G-PCC dimension ID of 2.

In this way, the mapping information indicates the G-PCC component ID and dimension ID corresponding to each dimension of each component of the Gaussian data. For example, for subcomponent C46 shown in Figure 42, G-PCC component ID = 18 and G-PCC dimension ID = 2 are indicated.

Note that while Figure 43 shows an example of mapping information indicating the subcomponents (dimensions) of the G-PCC that correspond to each subcomponent (dimension) of the Gaussian data, mapping information indicating the subcomponents of the Gaussian data that correspond to each subcomponent of the G-PCC may also be used.

Furthermore, Figure 43 shows an example of mapping information indicating the correspondence between Gaussian data subcomponents and G-PCC subcomponents, but if there are no restrictions on the number of subcomponents in a G-PCC component, mapping information indicating the correspondence between Gaussian components and G-PCC components may also be used. In other words, it is not necessary to indicate the correspondence between dimensions.

The conversion information indicates the parameters (conversion parameters) used when converting the component data. Figure 44 shows an example of the configuration of conversion information. For example, if a scale and offset are used in the conversion, the conversion information includes a scale value and an offset value.

For example, the conversion is performed using the scale value and offset value according to the following formula:

Converted data = Pre-conversion data x Scale value + Offset

The formula used for the conversion may be a predefined formula, or multiple conversion formulas may be defined and the formula used may be switched according to predetermined conditions. Information indicating the conversion formula used may also be stored in the bitstream.

In Figure 43, the transformation information is provided for each subcomponent (dimension) of the Gaussian data, but the transformation information may also be provided for each component of the Gaussian data. In other words, the transformation method may be changed on a component-by-component basis, or on a subcomponent-by-subcomponent basis.

Note that some of the syntax shown in FIG. 43 may be omitted. For example, if the order in which rotation, scale, and SH coefficient information is written and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, and number of Gaussian data dimensions may be omitted.

Furthermore, each attribute information is encoded based on the mapping information to the attribute component determined by the second pre-processing unit 332. Figure 45 shows an example of the structure of encoded data when Gaussian data is G-PCC encoded.

The SPS describes the G-PCC attribute component ID (attr_id), G-PCC attribute type (attribute_type), and number of dimensions of the G-PCC attribute component (num_dimension) for each G-PCC component. A new G-PCC attribute type may be defined here to indicate that each element of the Gaussian data is to be coded.

Figure 46 shows an example of G-PCC attribute types for Gaussian data. As shown in Figure 46, for example, G-PCC attribute types may be defined that indicate the rotation, scale, SH coefficient, and transparency of Gaussian data.

A G-PCC attribute component ID (attr_id) is assigned to the header of the ADU. The decoding device can identify which G-PCC attribute type each ADU has by using the G-PCC attribute component ID contained in the ADU header and the information contained in the SPS.

Furthermore, the mapping information (Gaussian data attribute mapping information SEI) may be transmitted as SEI or may be included in the SPS. For example, the SPS may include mapping information indicating the Gaussian components corresponding to the G-PCC components.

FIG. 47 is a flowchart of processing by a decoding device according to this embodiment. First, the decoding device obtains multiple TLV units from the bitstream, and then obtains multiple data units (GDU, ADU) and multiple parameter sets (SPS, GPS, APS) from the multiple TLV units (S101).

Next, the decoding device decodes the three-dimensional coordinates using the SPS, GPS, and GDU (S102). Next, the decoding device decodes the multiple attribute components described in the SPS using the APS, ADU, and three-dimensional coordinates (S103).

Next, the decoding device obtains mapping information by analyzing the Gaussian data attribute mapping information SEI (S104). Next, the decoding device reconstructs second Gaussian data from the decoded attribute components and the mapping information (S105). Next, the decoding device obtains transformation information and generates first Gaussian data by inversely transforming the second Gaussian data using the transformation information (S106).

[others]
In the above description, for simplicity, an example in which Gaussian data is one frame is shown, but this method can also be applied when there are multiple frames of Gaussian data. In this case, multi-frame coding in the G-PCC method or inter-prediction in the G-PCC method may be used.

The encoding device may also generate multiple pieces of divided data by dividing the Gaussian data into multiple regions, and encode each of the multiple pieces of divided data. The encoding device may also assign the divided data to tiles or slices in the G-PCC and encode them.

Note that while the above explanation shows an example of encoding attribute information (scale, rotation, SH coefficient, transmittance, etc.) of the second Gaussian data, the method of this embodiment can also be used to encode other attribute information.

FIG. 48 is a block diagram showing a system for encoding and decoding first Gaussian data. The system shown in FIG. 48 includes a second pre-processing unit 371, a G-PCC encoding unit 372, a G-PCC decoding unit 373, and a second post-processing unit 374. FIG. 49 is a block diagram showing an example configuration of the second pre-processing unit 371. The second pre-processing unit 371 includes a conversion unit 375 and a mapping unit 376.

For example, when encoding a covariance matrix associated with three-dimensional coordinates contained in the first Gaussian data, the conversion unit 375 included in the second pre-processing unit 371 converts the format of the covariance matrix. The mapping unit 376 generates converted first Gaussian data by mapping the format-converted covariance matrix to G-PCC attribute information. The G-PCC encoding unit 372 generates encoded data by encoding the converted first Gaussian data, the conversion information, and the mapping information.

The G-PCC decoding unit 373 decodes the encoded data to generate transformed first Gaussian data, transformation information, and mapping information. The second post-processing unit 374 uses the mapping information and transformation information to inversely transform the transformed first Gaussian data, thereby restoring the first Gaussian data. In other words, the covariance matrix is restored. In this way, the system can encode and decode first Gaussian data.

Furthermore, when converting a covariance matrix, the conversion unit 375 may convert a 3x3 covariance matrix into a format that can be encoded using G-PCC. For example, the conversion unit 375 divides the 3x3 covariance matrix into three three-dimensional components. Alternatively, if the covariance matrix is a symmetric matrix, the conversion unit 375 may divide the six-dimensional elements, excluding elements with the same value, into two three-dimensional components. Figure 50 is a diagram illustrating this conversion process. In the example shown in Figure 50, of the nine elements of the covariance matrix, six elements ([Sxx, Sxy, Sxz, Syy, Syz, Szz]) are extracted, excluding elements (Syz, Szx, Szy) that have the same value as other elements.

In addition, conversion information indicating the details of the conversion is generated. The conversion information is stored in the metadata. Note that if the conversion method is uniquely defined in advance, the conversion information does not need to be stored in the coded data (bitstream).

As described above, by using the processing and method described in this embodiment, it is possible to encode Gaussian data using G-PCC.

Specifically, the encoding device encodes the three-dimensional coordinates contained in the Gaussian data using position information encoding in G-PCC. The encoding device also encodes attribute information associated with the three-dimensional coordinates (scale, rotation, SH coefficient, transparency, etc.) using attribute information encoding in G-PCC.

The encoding device also converts each element of the Gaussian data's attribute information into a data format that can be encoded using G-PCC. The encoding device also stores conversion information indicating the details of the conversion in metadata. This allows the decoding device to reconstruct Gaussian data from the decoded data.

The encoding device also associates the components or dimensions of the Gaussian data with attribute components or dimensions, which are units that can be coded using G-PCC. This allows the encoding device to code the Gaussian data using the G-PCC coding method. Furthermore, by defining a G-PCC attribute type that corresponds to the Gaussian data, the encoding device can code the Gaussian data using the G-PCC coding method.

In addition, the encoding device stores mapping information in the metadata that indicates the correspondence between the components and dimensions of the Gaussian data and the attribute components and dimensions. This allows the decoding device to reconstruct Gaussian data from the decoded data.

(Second Aspect)
[Gaussian data encoding method]
Below, we describe a general data structure for encoding Gaussian data, as well as a data structure that allows Gaussian data to be encoded using G-PCC or other encoding methods.

By using the data structure below, a general-purpose method can be realized that can encode Gaussian data using a variety of 3D encoding methods or 3D codec standards. This makes it possible to encode Gaussian data using a variety of 3D codec methods, including standards created by various standards organizations, international standards, national standards, and standards established by any organization.

Figure 51 shows the protocol stack of an encoding standard for encoding Gaussian data. 3D Gaussian Codec is an encoding standard for encoding Gaussian data. Specific encoding methods for 3D Gaussian Codec include the V3C standard (ISO/IEC 23090-5), which encodes 3D data using one or more video codecs; the G-PCC standard (ISO-IEC 23090-9), which is a point cloud compression standard; and Draco, which is used for 3D mesh data. When the V3C standard is used, video codec standards such as HEVC or VVC (Versatile Video Coding) are also used.

FIG. 52 is a block diagram showing an example configuration of an encoding device 400 (three-dimensional data encoding device) according to this embodiment. The encoding device 400 generates a bit stream (encoded data) by encoding second Gaussian data. This encoding device 400 includes a second pre-processing unit 401, a G-PCC encoding unit 402, a V3C encoding unit 403, a metadata encoding unit 404, and a multiplexing unit 405.

The second pre-processing unit 401 converts and maps the second Gaussian data into data that can be encoded by the 3D codec used, depending on the 3D codec used.

For example, the second pre-processing unit 401 converts and maps the second Gaussian data into data that can be coded using the G-PCC coding method. The G-PCC coding unit 402 generates a G-PCC TLV unit by coding the converted data using the G-PCC coding method.

Furthermore, the second pre-processing unit 401 converts and maps the second Gaussian data into data that can be coded using the V3C coding method. The V3C coding unit 403 generates V3C units by coding the converted data using the V3C coding method.

The metadata encoding unit 404 generates metadata by encoding the conversion information and mapping information related to the conversion and mapping performed by the second pre-processing unit 401. For example, the metadata encoding unit 404 stores the conversion information and mapping information in the metadata.

The multiplexing unit 405 generates a bitstream by unitizing and multiplexing the G-PCC TLV unit or V3C unit and the metadata. Specifically, the multiplexing unit 405 stores the G-PCC TLV unit or V3C unit and the metadata in a 3D Data Unit, which is a data unit of the 3D Gaussian Codec Layer.

FIG. 53 is a block diagram showing an example configuration of a decoding device 410 (three-dimensional data decoding device) according to this embodiment. The decoding device 410 generates second Gaussian data by decoding a bitstream. This bitstream is generated, for example, by the encoding device 400 shown in FIG. 52. The decoding device 410 includes a demultiplexing unit 411, a G-PCC decoding unit 412, a V3C decoding unit 413, a metadata decoding unit 414, and a second post-processing unit 415.

The demultiplexing unit 411 demultiplexes and demultiplexes the 3D data units contained in the bitstream to generate G-PCC TLV units or V3C units and metadata.

The G-PCC decoder 412 generates decoded data by decoding the G-PCC TLV unit using the G-PCC decoding method. The V3C decoder 413 generates decoded data by decoding the V3C unit using the V3C decoding method.

The metadata decoding unit 414 generates conversion information and mapping information by decoding the metadata. For example, the metadata decoding unit 414 obtains the conversion information and mapping information from the metadata.

The second post-processing unit 415 inversely transforms the decoded data into second Gaussian data depending on the 3D codec used for decoding.

FIG. 54 is a flowchart of the encoding process performed by the encoding device 400. First, the encoding device 400 determines the codec type to be used for encoding (S201). For example, the codec type to be used for encoding may be specified by the user, or may be determined based on the contents of the Gaussian data.

If the codec type used for encoding is the G-PCC encoding method (G-PCC in S201), the encoding device 400 encodes the Gaussian data using the G-PCC encoding method and outputs a G-PCC TLV unit (S202).

On the other hand, if the codec type used for encoding is the V3C encoding method (V3C in S201), the encoding device 400 encodes the Gaussian data using the V3C encoding method and outputs a V3C unit (S203).

Next, the encoding device 400 stores information indicating the codec type used for encoding in common metadata (S204). Here, common metadata refers to metadata that can be used in common across multiple encoding methods (G-PCC encoding method and V3C encoding method).

Next, the encoding device 400 stores the conversion information and mapping information in the metadata (S205). Next, the encoding device 400 multiplexes the metadata and encoding unit (G-PCC TLV unit or V3C unit) into a 3D data unit that can be used in both the G-PCC encoding method and the V3C encoding method (S206).

Figure 55 is a flowchart of the decoding process performed by the decoding device 410. First, the decoding device 410 obtains a common 3D data unit, and then obtains data units and metadata from the 3D data unit (S211). Next, the decoding device 410 analyzes the metadata and determines the codec type used for encoding (the codec type to be used for decoding) based on information indicating the codec type included in the metadata (S212).

If the codec type used for encoding is the G-PCC encoding method (G-PCC in S213), the decoding device 410 acquires the TLV unit and decodes it using the G-PCC decoding method to generate a component (S214). On the other hand, if the codec type used for encoding is V3C (V3C in S213), the decoding device 410 acquires the V3C unit and decodes it using the V3C decoding method to generate a component (S215).

Next, the decoding device 410 reconstructs Gaussian data using the decoded components and the mapping information (S216). Next, the decoding device 410 obtains the transformation information and generates second Gaussian data by inversely transforming the Gaussian data using the transformation information (S217).

Figure 56 shows the basic data structure of a Gaussian codec. The 3D data unit in the 3D Gaussian codec layer is a common unit in Gaussian codecs.

Figure 57 is a diagram showing an example of the structure of a 3D data unit (3DDataUnit). The payload (3DDU_data) of the 3D data unit stores various data units that make up the Gaussian codec. The header (3DDU_header) of the 3D data unit contains a 3D data unit type (3DDataUnitType) that indicates the type of data stored in the payload. Examples of 3D data unit types (3DDataUnitType) include (1) V3C unit, (2) G-PCC TLV unit, (3) Draco unit, (4) 3D parameter set (3DPS), which is a parameter set common to multiple codecs, (5) Gaussian sequence parameter set (GSSPS), which is a parameter set including metadata common to a sequence, (6) Gaussian frame parameter set (GSFPS), which is a parameter set including metadata common to a frame (a 3D model at a certain time or time interval), and (7) SEI (Gaussian GSSEI).

Furthermore, if the 3D data unit type (3DDataUnitType) is a V3C unit, the payload of the V3C unit includes an ACL_NAL unit or a VCL_NAL unit. The VCL_NAL unit belongs to the video codec layer and is defined in two-dimensional video compression standards such as HEVC or VVC. The ACL_NAL unit belongs to the atlas codec layer. The ACL_NAL unit stores data for converting two-dimensional images back into three-dimensional data.

In addition, the 3D data unit may also store a Draco unit based on standards such as Draco.

Figure 58 is a diagram showing an example of the configuration of a 3DPS (3DParameterset), which is a common parameter set used in Gaussian encoding. 3DPS includes information (profile_level) indicating the profile and level that indicate the combination of the encoding standard, encoding method, and toolset used to encode Gaussian data, as well as Gaussian component information (3D_codec_type). This information includes information indicating the profile (profile), information indicating the level (level), and information indicating the encoding standard and encoding method (3D_codec_type).

3D_codec_type indicates the encoding standard used to encode Gaussian data. For example, 3D_codec_type = 0 indicates that the V3C encoding method was used for encoding, and 3D_codec_type = 1 indicates that the G-PCC encoding method was used for encoding.

The decoding device can determine the decoding method by analyzing 3D_codec_type. The decoding device may switch the decoding method based on 3D_codec_type, or may decide not to decode if it does not support the decoding function for the encoding method indicated by 3D_codec_type. Note that the profile and level may also include information indicating the encoding standard.

Furthermore, restrictions may be placed on the 3D data unit type (3DDataUnitType) depending on the encoding method. For example, if the encoding method is G-PCC, the use of V3C units may be prohibited, and if the encoding method is V3C, the use of G-PCC TLV units may be prohibited.

Gaussian data component information includes information indicating the configuration of Gaussian components. Figure 59 is a diagram showing an example configuration of Gaussian data component information. Gaussian data component information includes the number of Gaussian data components, which indicates the number of Gaussian components.

Gaussian data component information includes, for each Gaussian component, a Gaussian component ID, a Gaussian data type, and the number of Gaussian data dimensions. The Gaussian component ID is identification information for uniquely identifying a Gaussian component. The Gaussian data type indicates the type of Gaussian component (rotation, scale, SH coefficient, transmittance, etc.). The number of Gaussian data dimensions indicates the number of dimensions (number of subcomponents) of the Gaussian component.

The Gaussian data component information also includes mapping information that depends on the codec. For example, if the codec type (3D_codec_type) is G-PCC, the Gaussian data component information includes G-PCC mapping information (gpcc_mapping_info) that indicates the correspondence between the Gaussian data components and the G-PCC codec components.

Figure 60 shows an example of the configuration of G-PCC mapping information. The G-PCC mapping information includes a G-PCC component ID, a G-PCC dimension ID, and conversion information as information for each dimension of the Gaussian component.

The G-PCC subcomponent ID is an identifier that indicates the G-PCC component number. The G-PCC dimension ID is an identifier that indicates the subcomponent number of the G-PCC component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension in question.

Furthermore, if the codec type (3D_codec_type) is V3C, the Gaussian data component information includes V3C mapping information that indicates the correspondence between the Gaussian data components and the V3C codec components.

Figure 61 shows an example of the configuration of V3C mapping information. The V3C mapping information includes a V3C component ID, a V3C dimension ID, and conversion information as information for each dimension of the Gaussian component.

The V3C subcomponent ID is an identifier that indicates the V3C component number. The V3C dimension ID is an identifier that indicates the subcomponent number of the V3C component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension.

In addition, Gaussian data component information may include geometry component information in addition to attribute component information.

Furthermore, while an example of mapping information indicating G-PCC components corresponding to Gaussian components has been shown here, mapping information indicating Gaussian components corresponding to G-PCC components may also be used. Further, while an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC has been shown here, mapping information indicating the correspondence between Gaussian components and G-PCC components may also be used. In other words, the correspondence between subcomponents does not have to be shown.

Furthermore, some of the syntax shown here may be omitted. For example, if the order in which rotation, scale, and SH coefficient information is written and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, and number of Gaussian data dimensions may be omitted.

Below, another example of a data unit is described. Instead of storing Gaussian data in either a V3C unit or a G-PCC TLV unit, a data unit specific to Gaussian data (Gaussian data unit) may be defined.

Figure 62 is a diagram showing an example of the configuration of a Gaussian data unit. For example, the header of a Gaussian data unit includes a Gaussian DataUnit Type, which corresponds to the 3D data unit type (3DDataUnitType) described above. The Gaussian data unit type indicates the type of data stored in the payload of the Gaussian data unit. For example, the Gaussian data unit type includes a geometry data unit and an attribute data unit. The payloads of the geometry data unit and attribute data unit store encoded geometry data and attribute data, respectively.

The header of an attribute data unit contains an attribute data unit type. The attribute data unit type indicates the type of data stored in the payload of the attribute data unit. The attribute data unit type indicates an element of attribute information, and includes, for example, a rotation data unit (Rotation Data Unit), a scale data unit (Scale Data Unit), a color coefficient data unit (Color Coeff Data Unit), an alpha data unit (Alpha Data Unit), etc. The payloads of the rotation data unit, scale data unit, color coefficient data unit, and alpha data unit store encoded rotation, scale, SH coefficient, and transparency, respectively.

Note that while the above explanation has been given using an example of encoding 3D Gaussian data, a similar technique may be applied to data other than 3D Gaussian data. For example, a similar technique may be used for 3D models or 3D data using AI.

Furthermore, while the above explanation mainly focuses on examples in which the encoding methods are V3C and G-PCC, encoding methods other than V3C and G-PCC may also be used. Furthermore, the encoding method may be other than a 3D codec. For example, an image codec, a neural network codec, an audio codec, or the like may also be used as the encoding method.

Figure 63 shows a protocol stack for a standard for encoding 3D Gaussian data. A protocol stack or data structure for encoding 3D Gaussian data using at least one of a video base codec and a geometry base codec may be used as a method for encoding 3D Gaussian data.

Video-based codecs are image encoding methods that use video codecs, such as standards such as AVC, HEVC, or VVC standardized by MPEG, or standards such as VPCC (Video-based Point Cloud Compression), VCM (Video Coding for Machines), or VDMC (Video-based Dynamic Mesh Compression).

Geometry-based codecs include point cloud compression standards such as G-PCC or Draco, or mesh data compression standards. Note that the method for encoding 3D Gaussians may be expanded to include neural network-based codecs such as NNC (Neural Network Coding).

Instead of 3D Gaussian, a standard for encoding a 3D model may be defined. Figure 64 is a diagram showing a protocol stack for a standard for encoding a 3D model. A protocol stack and data structure for encoding a 3D model using at least one of V3C and G-PCC may be used.

The 3D model may include (1) a point cloud, (2) a mesh, (3) a 3D model created using NeRF (Neural Radiance Fields), and (4) a 3D model generated using Gaussian Splating, etc.

Furthermore, the encoding device may encode some of the components of the 3D model data using video encoding, and other components using geometry-based encoding. For example, the encoding device may encode the information contained in the Gaussian data that represents shape information, such as three-dimensional coordinates, rotation, and scale, using G-PCC, and encode the SH coefficients using a video-based encoding method. Here, geometry-based encoding may be suitable for sparse geometric structure shapes, while video-based encoding may be suitable for multidimensional SH coefficients. Therefore, in such cases, encoding efficiency can be improved.

Figure 65 is a diagram showing an example of the configuration of Gaussian component information in this case. In addition to the Gaussian component information shown in Figure 59, the Gaussian component information shown in Figure 65 includes a 3D codec type (3D_codec_type) for each Gaussian component. This makes it possible to set a 3D codec type for each component, thereby realizing the above function. Note that when a standard for encoding 3D models is used as shown in Figure 64, for example, similar information is stored in 3D model component information indicating information about the components of a 3D model, instead of Gaussian component information.

(Third Aspect)
Below, we will explain a method and data structure for extending the V3C standard to enable the application of general-purpose 3D codecs such as Gaussian data, as well as a method and data structure for extending the V3C standard to enable the application of other 3D codecs such as G-PCC.

By using the data structure below, it is possible to encode and decode Gaussian data using various V3C-based standards. It also makes it possible to realize a general-purpose method that can encode Gaussian data using various 3D encoding methods or 3D codec standards. This makes it possible to encode Gaussian data using various 3D codec methods, such as standards created by various standards organizations, international standards, national standards, or standards established by any organization.

Figure 66 shows the protocol stack of an encoding standard for encoding Gaussian data. The V3C encoding method (V3C codec) is a structure that can encode 3D data using one or more video codec standards, such as HEVC or VVC.

Here, the V3C encoding method is extended to define a Gaussian codec layer. Specifically, atlas data is extended, and the metadata required for reconstructing Gaussian data and encoding or decoding is stored in the atlas data. Here, the atlas data includes data for converting two-dimensional images into three-dimensional data in the V3C encoding method. In addition, the V3C encoding method is extended to enable the application of geometry-based 3D encoding standards (hereinafter also referred to as GPC) such as G-PCC and Draco. In the V3C encoding method, the atlas data includes data for converting decoded two-dimensional images or encoded three-dimensional data decoded using a 3D encoding standard into Gaussian data.

FIG. 67 is a block diagram showing an example configuration of an encoding device 500 (three-dimensional data encoding device) according to this embodiment. The encoding device 500 generates a bit stream (encoded data) by encoding second Gaussian data. This encoding device 500 includes a second pre-processing unit 501, a geometry-based encoding unit 502, a video-based encoding unit 503, a metadata encoding unit 504, and a multiplexing unit 505.

The second pre-processing unit 501 converts and maps the second Gaussian data into data that can be encoded by the 3D codec used, depending on the 3D codec used.

For example, the second pre-processing unit 501 converts and maps the second Gaussian data into data that can be coded using a geometry-based coding method (such as G-PCC or Draco). The geometry-based coding unit 502 generates a GCL NAL unit by coding the converted data using the geometry-based coding method. Note that when G-PCC is used, the geometry-based coding unit 502 may output a TLV unit.

The second pre-processing unit 501 also converts and maps the second Gaussian data into data that can be coded using a video-based coding method (such as AVC, HEVC, or VVC). The video-based coding unit 503 generates VCL NAL units by coding the converted data using the video-based coding method.

The metadata encoding unit 504 stores the conversion data and mapping information in metadata. Here, the metadata is metadata that extends the V3C standard. For example, an extension area of the V3C parameter set in the V3C standard may be used as this metadata. Alternatively, an extension of the Atlas sequence parameter set (ASPS) or Atlas frame parameter set (AFPS) may be used as this metadata. The metadata encoding unit 504 encodes the metadata to generate a V3C parameter set or an Atlas data unit (ACL NAL unit).

The multiplexing unit 505 generates a bitstream by unitizing and multiplexing the GCL NAL unit or VCL NAL unit and the V3C parameter set or ACL NAL unit. Specifically, the multiplexing unit 505 generates a bitstream by storing the GCL NAL unit or VCL NAL unit and the V3C parameter set or ACL NAL unit in a 3V3C unit.

FIG. 68 is a block diagram showing an example configuration of a decoding device 510 (three-dimensional data decoding device) according to this embodiment. The decoding device 510 generates second Gaussian data by decoding a bitstream. This bitstream is generated, for example, by the encoding device 500 shown in FIG. 67. The decoding device 510 includes a demultiplexing unit 511, a geometry-based decoding unit 512, a video-based decoding unit 513, a metadata decoding unit 514, and a second post-processing unit 515.

The demultiplexing unit 511 demultiplexes and demultiplexes the V3C units contained in the bitstream to generate GCL NAL units or VCL NAL units, and V3C parameter sets or ACL NAL units.

The geometry-based decoding unit 512 generates decoded data by decoding the GCL NAL units using a geometry-based decoding method (such as G-PCC or Draco). The video-based decoding unit 513 generates decoded data by decoding the VCL NAL units using a video-based decoding method (such as AVC, HEVC, or VVC).

The metadata decoding unit 514 generates conversion information and mapping information by decoding the V3C parameter set or ACL NAL unit. For example, the metadata decoding unit 514 obtains the conversion information and mapping information from the V3C parameter set or ACL NAL unit.

The second post-processing unit 515 inversely transforms the decoded data into second Gaussian data depending on the 3D codec used for decoding.

Figure 69 is a flowchart of the encoding process performed by the encoding device 500. First, the encoding device 500 determines the codec type (encoding method) to be used for encoding (S501). For example, the codec type to be used for encoding may be specified by the user, or may be determined based on the contents of the Gaussian data.

If the codec type used for encoding is a geometry-based encoding method (G-PCC or Draco) (geometry-based in S501), the encoding device 500 encodes the Gaussian data using the geometry-based encoding method (G-PCC or Draco) and outputs a GCL NAL unit (S502).

On the other hand, if the codec type used for encoding is a video-based encoding method (video-based in S501), the encoding device 500 encodes the Gaussian data using a video-based encoding method (video codec) and outputs a VCL NAL unit (S503).

Next, the encoding device 500 stores information indicating the codec type used for encoding in the metadata (S504).

Next, the encoding device 500 stores the conversion information and mapping information in the metadata (S505). Next, the encoding device 500 multiplexes the metadata and the encoding unit (GCL NAL unit or VCL NAL unit) into the V3C unit (S506).

Figure 70 is a flowchart of the decoding process performed by the decoding device 510. First, the decoding device 510 acquires multiple V3C units, and then acquires data units (GCL NAL units or VCL NAL units) and metadata from the multiple V3C units (S511). Next, the decoding device 510 analyzes the metadata and determines the codec type used for encoding (the codec type to be used for decoding) based on the information indicating the codec type included in the metadata (S512).

If the codec type used for encoding is a geometry-based encoding method (geometry-based in S513), the decoding device 510 acquires the GCL NAL unit and generates components by decoding the GCL NAL unit using a geometry-based decoding method (G-PCC or Draco) (S514). On the other hand, if the codec type used for encoding is a video-based encoding method (video-based in S513), the decoding device 510 acquires the VCL NAL unit and generates components by decoding the VCL NAL unit using a video-based decoding method (video codec) (S515).

Next, the decoding device 510 reconstructs Gaussian data using the decoded components and the mapping information (S516). Next, the decoding device 510 obtains the transformation information and generates second Gaussian data by inversely transforming the Gaussian data using the transformation information (S517).

Figure 71 shows an example of the configuration of a V3C unit that has been extended to enable the application of geometry-based 3D codecs to the V3C standard. Figure 72 shows an example of the configuration of a V3C unit. The V3C unit includes a header (V3C unit header) and a payload (V3C data unit).

The payload of a V3C unit stores ACL NAL units belonging to the atlas codec layer or video codec layer. Furthermore, the payload of an extended V3C unit stores GCL NAL units that store geometry-based 3D codec data.

The V3C unit header contains a V3C unit type that indicates the type of data contained in the payload. For example, V3C unit type = 10 indicates a geometry-based PCC unit (GCL NAL unit).

Figure 73 shows an example of the structure of a GCL NAL unit. A GCL NAL unit includes a header (GCL unit header) and a payload (GCL data unit). The payload of the GCL NAL stores a G-PCC TLV unit or a Draco unit.

The header of a GCL NAL unit stores a GCL unit type, which indicates the type of geometry-based 3D codec, which is the type of data contained in the payload. For example, GCL unit type = 0 indicates G-PCC point cloud compression (G-PCC TLV unit). GCL unit type = 1 indicates Draco point cloud compression (Draco unit).

Note that while the example described here shows a G-PCC TLV unit and the like being stored in a GCL NAL unit, a G-PCC TLV unit or a unit defined by each 3D codec method may also be stored directly in a V3C unit.

Figure 74 is a diagram showing an example of the configuration of an extended V3C parameter set (V3CParameterset). The V3C parameter set includes information indicating the profile and level (profile_level), which indicates the combination of the encoding standard, encoding method, and toolset used in encoding Gaussian data. This information includes information indicating the profile (profile), information indicating the level (level), and information indicating the codec used for encoding (is_video_codec and is_3d_codec).

Profile indicates the codec that uses an extension of V3C, such as VPCC or DMC. For example, profile indicates an extension of V3C's 3D Gaussian codec. For example, profile=0 indicates VPCC, profile=1 indicates DMC, and profile=2 indicates the 3D Gaussian codec.

is_video_codec is a flag indicating whether a video codec is used. If a video codec is used (is_video_codec = 1), the V3C parameter set includes a video codec type (video_codec_type) that indicates the video codec used.

is_3d_codec is a flag indicating whether a 3D codec is used. If a 3D codec is used (is_3d_codec = 1), the V3C parameter set includes a 3D codec type (3d_codec_type) that indicates the 3D codec used. For example, 3D codec type = 0 indicates G-PCC, and 3D codec type = 1 indicates Draco. Note that the predefined "4CC" code may also be used for the 3D codec type.

Note that if a 3D Gaussian codec is used and the values of is_video_codec and is_3d_codec are exclusive (one is 1 and the other is 0), a constraint may be specified stating that the bitstream conformance condition is that the values of is_video_codec and is_3d_codec are exclusive. In other words, the decoding device may determine that the bitstream is inappropriate (contains an error) if the values of is_video_codec and is_3d_codec are not exclusive.

When a 3D Gaussian codec is used (profile = 3D Gaussian codec), the V3C parameter set includes Gaussian component information (3d_codec_type) as extended information for the V3C 3D Gaussian codec.

Figure 75 shows an example of the configuration of Gaussian data component information. The Gaussian data component information includes the number of Gaussian data components, which indicates the number of Gaussian components.

Gaussian data component information includes, for each Gaussian component, a Gaussian component ID, a Gaussian data type, and the number of Gaussian data dimensions. The Gaussian component ID is identification information for uniquely identifying a Gaussian component. The Gaussian data type indicates the type of Gaussian component (rotation, scale, SH coefficient, transmittance, etc.). The number of Gaussian data dimensions indicates the number of dimensions (number of subcomponents) of the Gaussian component.

The Gaussian data component information also includes mapping information that depends on the codec. For example, if a video codec is used (is_video_codec = 1), the Gaussian data component information includes video mapping information (video_mapping_info) that indicates the correspondence between Gaussian data components and video codec components. If a 3D codec is used (is_3d_codec = 1), the Gaussian data component information includes 3D mapping information (3D_mapping_info) that indicates the correspondence between Gaussian data components and 3D codec components.

Figure 76 shows an example of the configuration of video mapping information. The video mapping information includes a video component ID, a video dimension ID, and transformation information as information for each dimension of the Gaussian component.

The video subcomponent ID is an identifier that indicates the video component number. The video dimension ID is an identifier that indicates the subcomponent number of the video component. The conversion information indicates the parameters (conversion parameters) used when converting the data of that dimension.

Figure 77 shows an example of the configuration of 3D mapping information. When G-PCC is used, the 3D mapping information includes the G-PCC component ID, G-PCC dimension ID, and transformation information as information for each dimension of the Gaussian component.

The G-PCC subcomponent ID is an identifier that indicates the G-PCC component number. The G-PCC dimension ID is an identifier that indicates the subcomponent number of the G-PCC component. The conversion information indicates the parameters (conversion parameters) used when converting the data of the dimension in question.

In addition, when Draco is used, the 3D mapping information includes information indicating the correspondence between the components of the Gaussian data and the components of the Draco codec.

In addition, Gaussian data component information may include geometry component information in addition to attribute component information.

Furthermore, while an example of mapping information indicating G-PCC components corresponding to Gaussian components has been shown here, mapping information indicating Gaussian components corresponding to G-PCC components may also be used. While an example of mapping information indicating the correspondence between subcomponents of Gaussian data and subcomponents of G-PCC has been shown here, mapping information indicating the correspondence between Gaussian components and G-PCC components may also be used. In other words, the correspondence between subcomponents does not need to be shown.

Furthermore, some of the syntax shown here may be omitted. For example, if the order in which the information about rotation, scale, and SH coefficients is written and the number of dimensions are predetermined, the Gaussian data component ID, Gaussian data type, and number of Gaussian data dimensions may be omitted.

Note that, although the above description has been given of an example in which the data to be encoded is 3D Gaussian data, the data to be encoded is not limited to 3D Gaussian data. For example, the protocol stack or data structure described in this embodiment may be used to encode a 3D model (3D model codec) generated by another method, such as NeRF. For example, the data to be encoded may be a 3D model (point cloud, mesh, or other 3D model) created using AI (Artificial Intelligence).

In addition to video-based and geometry-based standards, V3C encoding methods may also use neural network codecs trained through machine learning, etc.

Figure 78 shows an extended V3C protocol stack. For example, as shown in Figure 78, the V3C standard may be extended to enable transmission of neural networks (NNs). For example, a neural network-based coding method such as NNC (Neural Network Coding) is used for neural networks.

In addition, a codec that extends the V3C standard may be defined, and one or more of the following encoding methods may be used: video-based, geometry-based, and neural network-based encoding methods.

Furthermore, for example, when geometry information (position information) and attribute information are encoded in a point cloud compression method using AI, the geometry information may be encoded using learning. For example, a neural network-based encoding method or a geometry-based encoding method may be used for the geometry information, and a video-based encoding method may be used for the attribute information. This may potentially improve compression performance (encoding efficiency).

Figure 79 is a diagram showing an extended V3C protocol stack. For example, the V3C standard is extended as shown in Figure 79. For example, the V3C parameter set, ASPS, or AFPS is extended to define a data unit that stores AI-PCC. This makes it possible to use at least one of video-based, geometry-based, and neural network-based encoding methods as the codec method, enabling encoding and decoding of point cloud compression using AI.

[summary]
As described above, the encoding device (three-dimensional data encoding device) according to the embodiment performs the processing shown in FIG. 80 . FIG. 80 is a flowchart of the encoding processing by the encoding device. The encoding device encodes three-dimensional data (e.g., Gaussian data, point clouds, meshes, etc.) including one or more first components. The encoding device selects one of a plurality of encoding methods including a first encoding method (e.g., V3C, AVC, HEVC, VVC, etc.) that is a video-based three-dimensional data encoding method, and a second encoding method different from the first encoding method (S601). If the first encoding method is selected (first encoding method in S602), the encoding device generates first encoded data (e.g., VCL NAL units) by encoding the one or more first components using the first encoding method (S603). If the second encoding method is selected (second encoding method in S602), the encoding device generates second encoded data (e.g., GCL NAL units) by encoding the one or more first components using the second encoding method (S604). NAL unit) is generated (S604), first information (is_video_codec, is_3d_codec, video_codec_type, or 3d_codec_type) indicating the selected encoding method is stored in first metadata (for example, V3C parameter set, ASPS, or AFPS) that is an extension of the metadata of the first encoding method (S605), and a bitstream including the first metadata and the first encoded data or the second encoded data is generated (S606).

This allows the encoding device to selectively select from multiple encoding methods the encoding method to use for encoding 3D data. This potentially improves encoding efficiency. Furthermore, by extending the video-based 3D data encoding method, it is possible to easily build a system that can achieve the above processing.

For example, the encoding device further converts one or more first components into one or more second components corresponding to the selected encoding method, and if the first encoding method is selected, generates first encoded data by encoding the one or more second components using the first encoding method, and if the second encoding method is selected, generates second encoded data by encoding the one or more second components using the second encoding method, and the first metadata further includes second information (e.g., mapping information) indicating the correspondence between the one or more first components and the one or more second components.

This allows the encoding device to encode three-dimensional data using the selected encoding method by converting the data into a format compatible with the selected encoding method. Furthermore, the decoding device can convert multiple second components into multiple first components using second information included in the bitstream.

For example, the second encoding method is a geometry-based three-dimensional data encoding method (such as G-PCC or Draco). For example, the second encoding method is a neural network-based three-dimensional data encoding method (such as NNC). For example, the second encoding method is a three-dimensional data encoding method using AI.

For example, the three-dimensional data is Gaussian data. This allows the encoding device to selectively select an encoding method from multiple encoding methods to use for encoding the Gaussian data. This has the potential to improve encoding efficiency.

Figure 81 is a block diagram of the encoding device 10. For example, the encoding device 10 includes a processor 11 and a memory 12, and the processor 11 performs the above processing using the memory 12.

Furthermore, a decoding device (three-dimensional data decoding device) according to an embodiment performs the processing shown in FIG. 82. FIG. 82 is a flowchart of the decoding process by the decoding device. The decoding device decodes three-dimensional data (e.g., Gaussian data, point clouds, meshes, etc.) including one or more first components. The decoding device acquires a bitstream including first metadata (e.g., a V3C parameter set, ASPS, or AFPS) that is an extension of the metadata of a first encoding method (e.g., V3C, AVC, HEVC, or VVC, etc.) that is a video-based three-dimensional data encoding method, and encoded data (e.g., a VCL NAL unit or a GCL NAL unit) (S611), and obtains first information (is_video_codec, is _3d_codec, video_codec_type, or 3d_codec_type) is obtained from the first metadata (S612). If the first information indicates a first encoding method (first encoding method in S613), one or more first components are generated by decoding the encoded data using a first decoding method corresponding to the first encoding method (S614). If the first information indicates a second encoding method (second encoding method in S613), one or more first components are generated by decoding the encoded data using a second decoding method corresponding to the second encoding method (S615).

This allows the encoding device to selectively select from multiple encoding methods the encoding method to use for encoding 3D data. This potentially improves encoding efficiency. Furthermore, the decoding device can properly decode the encoded data generated by the encoding device. Furthermore, by expanding the video-based 3D data encoding method, a system capable of easily implementing the above processing can be constructed.

For example, the encoded data is generated by encoding one or more second components resulting from the conversion of one or more first components. The decoding device further acquires second information (e.g., mapping information) indicating the correspondence between the one or more first components and one or more second components from the first metadata, and if the first information indicates a first encoding method, generates one or more second components by decoding the encoded data using a first decoding method; if the first information indicates a second encoding method, generates one or more second components by decoding the encoded data using a second decoding method, and converts the one or more second components into one or more first components using the correspondence indicated in the second information.

This allows the encoding device to encode three-dimensional data using the selected encoding method by converting the data into a format compatible with the selected encoding method. Furthermore, the decoding device can convert multiple second components into multiple first components using second information included in the bitstream.

For example, the second encoding method is a geometry-based three-dimensional data encoding method (such as G-PCC or Draco). For example, the second encoding method is a neural network-based three-dimensional data encoding method (such as NNC). For example, the second encoding method is a three-dimensional data encoding method using AI.

For example, the three-dimensional data is Gaussian data. This allows the encoding device to selectively select an encoding method from multiple encoding methods to use for encoding the Gaussian data. This has the potential to improve encoding efficiency.

Figure 83 is a block diagram of the decoding device 20. For example, the decoding device 20 includes a processor 21 and a memory 22, and the processor 21 performs the above processing using the memory 22.

The above describes an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device) according to an embodiment and modifications of the present disclosure, but the present disclosure is not limited to these embodiments.

Furthermore, each processing unit included in the encoding device, decoding device, etc. according to the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented on a single chip, or some or all of them may be included on a single chip.

Furthermore, integrated circuits are not limited to LSIs, but may be realized using dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow the connections and settings of circuit cells within the LSI to be reconfigured, may also be used.

Furthermore, in each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program appropriate for that component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

The present disclosure may also be realized as an encoding method (three-dimensional data encoding method) or a decoding method (three-dimensional data decoding method) executed by an encoding device (three-dimensional data encoding device) and a decoding device (three-dimensional data decoding device), etc.

The present disclosure may also be realized as a program that causes a computer, processor, or device to execute the encoding method or decoding method. The present disclosure may also be realized as a bitstream generated by the encoding method. The present disclosure may also be realized as a recording medium on which the program or bitstream is recorded. For example, the present disclosure may also be realized as a non-transitory computer-readable recording medium on which the program or bitstream is recorded.

Furthermore, the division of functional blocks in the block diagram is one example; multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to other functional blocks. Furthermore, the functions of multiple functional blocks with similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

Furthermore, the order in which each step in the flowchart is executed is merely an example to specifically explain the present disclosure, and orders other than those described above may also be used. Furthermore, some of the steps may be executed simultaneously (in parallel) with other steps.

The encoding device, decoding device, etc. according to one or more aspects have been described above based on the embodiments, but the present disclosure is not limited to these embodiments. Various modifications conceivable by those skilled in the art to these embodiments, as well as configurations constructed by combining components from different embodiments, may also be included within the scope of one or more aspects, provided they do not deviate from the spirit of the present disclosure.

This disclosure can be applied to encoding devices and decoding devices.

10 Encoding device 11, 21 Processor 12, 22 Memory 20 Decoding device 101 Three-dimensional data encoding system 102 Three-dimensional data decoding system 103 Sensor terminal 104 External connection unit 111 Three-dimensional data generation system 112 Presentation unit 113 Encoding unit 114 Multiplexing unit 115 Input/output unit 116 Control unit 117 Sensor information acquisition unit 118 Three-dimensional data generation unit 121 Sensor information acquisition unit 122 Input/output unit 123 Demultiplexing unit 124 Decoding unit 125 Presentation unit 126 User interface 127 Control unit 130 First encoding unit 131 Position information encoding unit 132 Attribute information encoding unit 133 Additional information encoding unit 134 Multiplexing unit 140 First decoding unit 141 Demultiplexing unit 142 Position information decoding unit 143 Attribute information decoding unit 144 Additional information decoding unit 150 Second encoding unit 151 Additional information generation unit 152 Position image generation unit 153 Attribute image generation unit 154 Video encoding unit 155 Additional information encoding unit 156 Multiplexing unit 160 Second decoding unit 161 Demultiplexing unit 162 Video decoding unit 163 Additional information decoding unit 164 Position information generation unit 165 Attribute information generation unit 171 Octree encoding unit 172 Prediction tree encoding unit 173 Octree decoding unit 174 Prediction tree decoding unit 181, 191 Octree generation unit 182, 192 Geometric information calculation unit 183, 193 Encoding table selection unit 184 Entropy encoding unit 194 Entropy decoding unit 201 LoD attribute information encoding unit 202 Transformed attribute information encoding unit 203 LoD attribute information decoding unit 204 Transformed attribute information decoding unit 211 Sorting unit 212 Haar transform unit 213 Quantization unit 214, 222 Inverse quantization unit 215, 223 Inverse Haar transform unit 216, 224 Memory 217 Arithmetic coding unit 221 Arithmetic decoding unit 231 Data division unit 232 Encoding unit 233 Decoding unit 234 Data combination unit 241 Encoding unit 242 TLV storage unit 301 Sensor information input unit 302 Three-dimensional data generation unit 303, 305 Rendering unit 304 Spherical harmonic function 311 Encoding unit 312 Multiplexing unit 321 Demultiplexing unit 322 Decoding unit 323 Application unit 324 Input interface unit 325 Rendering unit 326 Presentation unit 330, 400 Encoding device 331 First pre-processing unit 332, 371, 401 Second pre-processing unit 333, 372, 402 G-PCC encoding unit 341 Position information encoding unit 342 Rotation encoding unit 343 Scale encoding unit 344 SH coefficient encoding unit 345 Transmittance encoding unit 346 Metadata encoding unit 347, 405 Multiplexing unit 350, 410 Decoding device 351, 373, 412 G-PCC decoding unit 352, 374, 415 Second post-processing unit 353 First post-processing unit 361, 411 Demultiplexing unit 362 Position information decoding unit 363 Rotation decoding unit 364 Scale decoding unit 365 SH coefficient decoding unit 366 Transmittance decoding unit 367 Metadata decoding unit 375 Conversion unit 376 Mapping unit 403 V3C encoding unit 404 Metadata encoding unit 413 V3C decoding unit 414 Metadata decoding unit 500 Encoding device 501 Second pre-processing unit 502 Geometry-based encoding unit 503 Video-based encoding unit 504 Metadata encoding unit 505 Multiplexing unit 510 Decoding device 511 Demultiplexing unit 512 Geometry-based decoding unit 513 Video-based decoding unit 514 Metadata decoding unit 515 Second post-processing unit

Claims (14)

1. A method for encoding three-dimensional data, comprising: encoding three-dimensional data including one or more first components, the method comprising:
selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method that is different from the first encoding method;
If the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method;
If the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method;
storing first information indicating the selected encoding method in first metadata obtained by extending the metadata of the first encoding method;
A three-dimensional data encoding method for generating a bitstream including the first metadata and the first encoded data or the second encoded data. The three-dimensional data encoding method further comprises:
converting the one or more first components into one or more second components corresponding to the selected encoding format;
If the first encoding method is selected, generating the first encoded data by encoding the one or more second components using the first encoding method;
If the second encoding method is selected, generating the second encoded data by encoding the one or more second components using the second encoding method;
The three-dimensional data encoding method according to claim 1 , wherein the first metadata further includes second information indicating a correspondence between the one or more first components and the one or more second components. The three-dimensional data encoding method according to claim 1 , wherein the second encoding method is a geometry-based three-dimensional data encoding method. The three-dimensional data encoding method according to claim 1 , wherein the second encoding method is a neural network-based three-dimensional data encoding method. The three-dimensional data encoding method according to claim 1 , wherein the second encoding method is a three-dimensional data encoding method using AI. The three-dimensional data encoding method according to claim 1 , wherein the three-dimensional data is Gaussian data. 1. A method for decoding three-dimensional data, comprising: decoding three-dimensional data including one or more first components, the method comprising:
obtaining a bitstream including first metadata in which metadata of a first encoding method that is a video-based three-dimensional data encoding method is extended, and encoded data;
obtaining, from the first metadata, first information indicating a coding method used to code the one or more first components, from a plurality of coding methods including the first coding method and a second coding method different from the first coding method;
If the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method;
a three-dimensional data decoding method for generating the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method when the first information indicates the second encoding method. the encoded data is generated by encoding one or more second components obtained by transforming the one or more first components,
The three-dimensional data decoding method further includes:
acquiring second information indicating a correspondence relationship between the one or more first components and the one or more second components from the first metadata;
If the first information indicates the first encoding method, generating the one or more second components by decoding the encoded data using the first decoding method;
If the first information indicates the second encoding method, generating the one or more second components by decoding the encoded data using the second decoding method;
The three-dimensional data decoding method according to claim 7 , further comprising: converting the one or more second components into the one or more first components using the correspondence indicated by the second information. The three-dimensional data decoding method according to claim 7 , wherein the second encoding method is a geometry-based three-dimensional data encoding method. The three-dimensional data decoding method according to claim 7 , wherein the second encoding method is a neural network-based three-dimensional data encoding method. The three-dimensional data decoding method according to claim 7 , wherein the second encoding method is a three-dimensional data encoding method using AI. The three-dimensional data decoding method according to claim 7 , wherein the three-dimensional data is Gaussian data. 1. A three-dimensional data encoding device for encoding three-dimensional data including one or more first components, comprising:
a processor;
a memory;
The processor uses the memory to:
selecting one of a plurality of encoding methods including a first encoding method that is a video-based three-dimensional data encoding method and a second encoding method that is different from the first encoding method;
If the first encoding method is selected, generating first encoded data by encoding the one or more first components using the first encoding method;
If the second encoding method is selected, generating second encoded data by encoding the one or more first components using the second encoding method;
storing first information indicating the selected encoding method in first metadata obtained by extending the metadata of the first encoding method;
a three-dimensional data encoding device that generates a bitstream including the first metadata and the first encoded data or the second encoded data; 1. A three-dimensional data decoding device for decoding three-dimensional data including one or more first components, comprising:
a processor;
a memory;
The processor uses the memory to:
obtaining a bitstream including first metadata in which metadata of a first encoding method that is a video-based three-dimensional data encoding method is extended, and encoded data;
obtaining, from the first metadata, first information indicating a coding method used to code the one or more first components, from a plurality of coding methods including the first coding method and the second coding method;
If the first information indicates the first encoding method, generating the one or more first components by decoding the encoded data using a first decoding method corresponding to the first encoding method;
a three-dimensional data decoding device that, when the first information indicates the second encoding method, generates the one or more first components by decoding the encoded data using a second decoding method corresponding to the second encoding method.