WO2025152015A1 - Coding method, decoding method, code stream, coder, decoder, and storage medium - Google Patents

Abstract

Disclosed in the embodiments of the present application are a coding method, a decoding method, a code stream, a coder, a decoder, and a storage medium, which can improve decoding accuracy, thereby improving decoding performance. The decoding method comprises: parsing a code stream to determine a first base mesh and texture coordinate information of the first base mesh; on the basis of the first base mesh, performing mesh subdivision and deformation to determine a reconstructed deformed mesh; and on the basis of the texture coordinate information of the first base mesh and the reconstructed deformed mesh, determining texture coordinate information of the reconstructed deformed mesh.

Description

Coding and decoding method, code stream, encoder, decoder and storage medium Technical Field

The present application relates to the field of video coding and decoding technology, and in particular to a coding and decoding method, a bit stream, an encoder, a decoder and a storage medium.

Background Art

Video-based Dynamic Mesh Coding (VDMC) is a standard for compressing three-dimensional meshes. It mainly compresses three-dimensional meshes by leveraging the existing Visual Volumetric Video-based Coding (V3C) standard. Since three-dimensional meshes have encoding of connection information, the specific encoding process is slightly different from V3C.

However, during the encoding and decoding process of 3D meshes, the texture coordinates generated by mesh parameterization do not match the reconstructed deformed mesh, which reduces the decoding accuracy and affects the decoding performance.

Summary of the invention

The present application provides a coding and decoding method, a bit stream, an encoder, a decoder and a storage medium, which can improve decoding accuracy and improve decoding performance.

The technical solution of this application can be implemented as follows:

In a first aspect, an embodiment of the present application provides a decoding method, which is applied to a decoder, and the method includes:

Parse the code stream to determine the first basic grid and texture coordinate information of the first basic grid;

Performing mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh;

The texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh.

In a second aspect, an embodiment of the present application provides an encoding method, which is applied to an encoder, and the method includes:

Determine the base grid corresponding to the current image;

Based on the base mesh, determining texture coordinate information;

The basic grid and the texture coordinate information are encoded, and the obtained encoding bits are written into a bit stream.

In a third aspect, an embodiment of the present application provides a code stream, which is generated by bit encoding according to information to be encoded; wherein the information to be encoded includes at least one of the following:

Base mesh and texture coordinate information.

In a fourth aspect, an embodiment of the present application provides an encoder, comprising

The grid information determination part is configured to determine a basic grid corresponding to the current image; and determine texture coordinate information based on the basic grid;

The encoding part is configured to encode the basic grid and the texture coordinate information, and write the obtained encoding bits into a bit stream.

In a fifth aspect, an embodiment of the present application provides an encoder, including a first memory and a first processor, wherein:

A first memory, for storing a computer program that can be run on the first processor;

The first processor is used to execute the method described in the second aspect when running a computer program.

In a sixth aspect, an embodiment of the present application provides a decoder, comprising

A parsing part, configured to parse the code stream to determine the first basic grid and texture coordinate information of the first basic grid;

A subdivision and deformation part is configured to perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh;

The coordinate determination part is configured to determine the texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh.

In a seventh aspect, an embodiment of the present application provides a decoder, including a second memory and a second processor, wherein:

A second memory for storing a computer program that can be run on a second processor;

The second processor is used to execute the method described in the first aspect when running a computer program.

In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it implements the method as described in the first aspect, or implements the method as described in the second aspect.

The embodiments of the present application provide a coding and decoding method, a bit stream, an encoder, a decoder and a storage medium. At the decoding end, the bit stream is parsed to determine a first basic grid and texture coordinate information of the first basic grid; grid subdivision and deformation are performed based on the first basic grid to determine a reconstructed deformed grid; based on the texture coordinate information of the first basic grid and the reconstructed deformed grid, the texture coordinate information of the reconstructed deformed grid is determined. In this way, the texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh obtained by subdivision deformation, thereby reducing the distortion of the texture coordinates of the vertices and improving the accuracy of the texture coordinates and the matching degree with the reconstructed deformed mesh, thereby improving the quality of the reconstructed deformed mesh and improving the decoding performance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1A is a schematic diagram of a three-dimensional grid image 1;

FIG1B is a partial enlarged schematic diagram of a three-dimensional grid image;

FIG2 is a schematic diagram of a connection method of a three-dimensional grid;

FIG3 is a schematic diagram of a three-dimensional mesh coding framework based on subdivision deformation;

FIG4 is a schematic diagram of an implementation process of a grid simplification operation provided in an embodiment of the present application;

FIG5 is a schematic diagram of an implementation process of displacement vector calculation provided in an embodiment of the present application;

FIG6 is a schematic diagram of an implementation process of a subdivision operation provided in an embodiment of the present application;

FIG7 is a schematic diagram of an implementation process of texture map conversion provided by an embodiment of the present application;

FIG8 is a schematic diagram of a three-dimensional mesh decoding framework based on subdivision deformation;

FIG9 is a schematic diagram of the position change of the base mesh vertices provided in an embodiment of the present application;

FIG10 is a schematic diagram of a mesh shape comparison between a basic subdivided mesh and a reconstructed deformed mesh provided in an embodiment of the present application;

FIG11 is a schematic diagram of a mesh architecture of a codec provided in an embodiment of the present application;

FIG12 is a flowchart diagram 1 of a decoding method provided in an embodiment of the present application;

FIG13 is a schematic diagram of a three-dimensional grid decoding framework provided in an embodiment of the present application;

FIG14 is a second schematic diagram of a three-dimensional grid decoding framework provided in an embodiment of the present application;

FIG15 is a flowchart diagram 1 of a decoding method provided in an embodiment of the present application;

FIG16 is a flowchart diagram 1 of a decoding method provided in an embodiment of the present application;

FIG17 is an optional schematic diagram of at least one candidate location point provided in an embodiment of the present application;

FIG18 is a schematic diagram of an optional search range provided in an embodiment of the present application;

FIG19 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

FIG20 is a schematic diagram of a three-dimensional grid coding framework provided in an embodiment of the present application;

FIG21 is a second schematic diagram of a three-dimensional grid coding framework provided in an embodiment of the present application;

FIG22 is a schematic diagram of the composition structure of an encoder provided in an embodiment of the present application;

FIG23 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

FIG24 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

FIG25 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application;

FIG. 26 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

It should be noted that different data format bitstreams can be decoded and synthesized in the same video scene. Among them, at least image format, point cloud format, and mesh format can be included. In this way, real-time immersive video interaction services can be provided for multiple data formats (for example, mesh, point cloud, image, etc.) with different sources.

In the embodiment of the present application, the data format-based method can allow independent processing at the bitstream level of the data format. That is, like tiles or slices in video encoding, different data formats in the scene can be encoded in an independent manner, so that independent encoding and decoding can be performed based on the data format.

Generally speaking, 3D animation content uses a keyframe-based representation method, that is, each frame is a static mesh. Static meshes at different times have the same topological structure and different geometric structures. However, the amount of data of the 3D dynamic mesh represented based on keyframes is particularly large, so how to effectively store, transmit and draw it has become a problem faced by the development of 3D dynamic meshes. In addition, the spatial scalability of the mesh needs to be supported for different user terminals (computers, notebooks, portable devices, mobile phones); different mesh bandwidths (broadband, narrowband, wireless) need to support the quality scalability of the mesh. Therefore, 3D dynamic mesh compression is a very critical issue. Among them, "one frame" can be understood as an image. For example, a keyframe can be understood as a key image in a 3D animation.

A 3D grid is a 3D object surface composed of numerous polygons in space. A polygon consists of vertices and edges. FIG1A shows a 3D grid image, and FIG1B shows a partially enlarged schematic diagram of a 3D grid image. It can be seen from FIG1A and FIG1B that the grid surface is composed of closed polygons.

A two-dimensional image has information expressed at each pixel point and is distributed regularly, so there is no need to record its position information additionally; however, the distribution of vertices in the mesh in three-dimensional space is random and irregular, and the way polygons are formed requires additional regulations. Therefore, it is necessary to record the position of each vertex in space and the connection information of each polygon in order to fully express a mesh image. As shown in Figure 2, the same number of vertices and vertex positions, due to different connection methods, form completely different surfaces.

In addition to the above information, since the three-dimensional grid image is usually encoded using an existing two-dimensional image/video encoding method, the three-dimensional grid needs to be converted from three-dimensional space to a two-dimensional image, and the UV coordinates define this conversion process.

Similar to two-dimensional images, each position in the acquisition process may have corresponding attribute information, usually RGB color values, which reflect the color of the object; for three-dimensional meshes, in addition to color, the attribute information corresponding to each vertex is also commonly the reflectance value, which reflects the surface material of the object. The attribute information of the three-dimensional mesh is stored in a two-dimensional image, and its mapping from two-dimensional to three-dimensional is specified by UV coordinates. Among them, the three-dimensional mesh data usually includes three-dimensional geometric coordinate information (x, y, z), geometric connection relationship, UV coordinates, attribute map, etc.

It should also be noted that Video-based Dynamic Mesh Coding (VDMC) is a standard developed by the Moving Pictures Experts Group (MPEG) for compressing three-dimensional meshes. Its main idea is to compress three-dimensional meshes by utilizing the existing Visual Volumetric Video-based Coding (V3C) standard. Since there is connection information in the three-dimensional mesh that needs to be encoded, its specific encoding process is slightly different from V3C. It is necessary to expand the syntax and semantics of the V3C standard decoding end and the decoding operation to support the decoding and reconstruction of three-dimensional meshes.

FIG. 3 is a schematic diagram of a three-dimensional grid encoding framework, and FIG. 8 is a schematic diagram of a three-dimensional grid decoding framework.

At present, the overall framework of a VDMC encoder is shown in Figure 3. For the input mesh, the basic mesh and the corresponding subdivided deformed mesh are first obtained through the basic mesh generation module. Among them, the input mesh first obtains a simplified mesh through the simplification module, and then generates new texture coordinates for the simplified mesh through mesh parameterization. Then, the parameterized mesh is subdivided and deformed, that is, new vertices are inserted into the mesh according to a specific subdivision method and the distance from the vertex of the subdivided mesh to the nearest neighbor of the input mesh is calculated, which is called displacement, and the corresponding subdivided deformed mesh is obtained. Subsequently, the parameterized mesh, that is, the vertex position of the mesh before subdividing the deformation is adjusted according to the displacement information, and the adjusted mesh is called the basic mesh, which is sent to the basic mesh encoding module and compressed using the existing mesh encoder. In the inter-frame mode, motion vectors can also be generated for each vertex of the basic mesh according to the reference frame, and the basic mesh module only needs to compress the motion vector.

The base mesh is reconstructed after encoding, and then the displacement is calculated based on the reconstructed base mesh and the subdivided deformed mesh obtained in the base mesh generation stage. Subsequently, the displacement information is transformed, quantized, and the processed displacement can be encoded through a video encoder or an entropy encoder. Then, the reconstructed displacement information is applied to the subdivided base mesh to obtain a reconstructed subdivided deformed mesh, and the mesh, the original input mesh, and its corresponding texture map are input into the corresponding texture map conversion module to obtain the texture map corresponding to the reconstructed mesh, and the texture map is also encoded using a video encoder.

Next, the main modules in the encoder are described.

1. Basic mesh generation module:

The basic mesh generation module includes a mesh simplification module, a mesh parameterization module, and a subdivision deformation module. Each module is introduced below.

(1) Mesh simplification module:

Mesh simplification is to simplify the current input mesh into a basic mesh with relatively few points and faces, and keep the shape of the original mesh as much as possible. The focus of mesh simplification is the simplification operation and the corresponding merge error energy function. A feasible mesh simplification operation is shown in Figure 4, which merges the vertices at both ends of the edge into one vertex and deletes the connection between the two vertices. Repeat this process in the entire mesh according to certain rules to reduce the number of faces and vertices of the mesh to the target value.

(2) Grid parameterization module:

The mesh parameterization module is used to generate the corresponding texture coordinates for the mesh. Currently, there are many algorithms for parameterizing the mesh, such as the Isochart algorithm, the orthogonal projection algorithm, etc. In this coding framework, both of the above schemes can be used to parameterize the reconstructed base mesh.

Among them, the Isochart algorithm uses spectral analysis to achieve stretch-driven 3D mesh parameterization, UV unfolding, slicing and packing the 3D mesh into a 2D texture domain. A stretch threshold is set, and the algorithm is outlined as follows:

a) Compute surface spectral analysis to provide an initial parameterization

b) Perform stretch optimization iterations

c) If the stretch of this derived parameterization is less than a threshold, stop

d) Perform surface spectral clustering to divide the surface into charts

e) Use graph cut algorithm to optimize chart boundaries

f) Iterate the splitting of charts until the stretching criteria are met

Among them, the orthogonal projection algorithm may include the orthoAtlas algorithm, which is a projection-based mesh parameterization method that generates texture coordinates for the mesh through orthogonal projection. Its main process includes:

a) Calculate mesh properties, including the neighboring faces of each face and the area and normal vector of each face;

b) Determine the projection plane of each face based on the normal vector;

c) Start clustering all faces according to the projection plane to form a connected region, and first select the starting face of the cluster;

d) Iterating from the starting face, determining whether the adjacent faces of the face added to the connected region can be added to the connected region;

e) After each connected region is iterated, multiple connected regions are obtained;

f) determining whether to merge adjacent connected regions based on the error metric;

g) Detect whether there are overlapping areas during projection, remove the overlapping surfaces and regenerate the connected areas;

h) Arrange all the projected regions into a two-dimensional image.

(3) Subdivision deformation module:

The basic idea of the subdivision deformation module is shown in Figure 5, where the same concept is applied to the input 3D mesh to generate displacements. In Figure 5, the input 2D curve (represented by a 2D polyline), called the "original" curve, is first downsampled to generate a basic curve/polyline, called the "simplified" curve. The subdivision scheme is then applied to the simplified polyline to generate the "subdivided" curve. The subdivided polyline is then deformed to get a better approximation of the original curve. That is, a displacement vector is calculated for each vertex of the subdivided mesh so that the shape of the displaced curve is as close to the shape of the original curve as possible. These displacement vectors are the displacement information output by the module.

The subdivision deformation module takes the parameterized mesh as the input mesh and first subdivides the input mesh. The subdivision scheme can be arbitrarily selected. One possible scheme is the midpoint subdivision scheme, which subdivides each triangle into four subtriangles in each subdivision iteration, as shown in Figure 6. A new vertex is introduced in the middle of each edge. The subdivision of geometric information and attribute information is performed independently because the connection relationship between geometric information and attribute information is usually different.

For the subdivided mesh, find the nearest neighbor of each point on the original input mesh (including the points on the original mesh surface), which can be accelerated by data structures such as kdTree. The displacement of the geometric coordinates of each vertex of the subdivided mesh is obtained by calculating the distance between each vertex on the subdivided mesh and its nearest neighbor on the original input mesh. This module can obtain the base mesh and the corresponding subdivided deformed mesh.

2. Basic grid compression module:

There are three different compression modes for basic mesh compression, namely intra-frame mode, inter-frame mode and skip mode. In intra-frame mode, the input of the basic mesh compression module is a three-dimensional mesh, which contains geometric coordinates, connection relationships and attribute information associated with vertices, and is encoded through the existing static mesh encoder. In inter-frame mode, the basic mesh first uses RDO (rate-distortion trade-off) to decide whether to use skip mode. If skip mode is used, the reference mesh in the mesh buffer is directly selected as the basic mesh of the current image. If skip mode is not used, the motion vector of the vertex is calculated based on the reference image in the mesh buffer and the current image, and the corresponding motion vector is encoded. After encoding, the encoded mesh needs to be reconstructed so that it can be provided to subsequent modules for processing.

3. Displacement generation module and displacement encoding module:

The base mesh compression module compresses and reconstructs the base mesh, and the displacement generation module calculates the corresponding vertex displacement using the reconstructed base mesh and the subdivided deformed mesh obtained in the subdivided deformation step. At the same time, due to the existence of subdivided deformation, the displacement encoding module can encode the displacement in the LoD (Level of Detail) manner, that is, each level is encoded independently to support the decoding end to decode according to the required level. After obtaining the layered displacement coefficients, the displacement coefficients of each level are encoded independently.

There are many ways to encode displacement: one way is to consider converting the coordinate system of the displacement, that is, converting the coordinate system of the displacement of each vertex into a coordinate system constructed by the normal vector of the vertex corresponding to the displacement and the two components tangent to the normal vector. Then transform the displacement, such as wavelet transform. Quantize the transformed coefficients and arrange them in the image in the order of scanning, and apply video encoding to the image. In addition, the generated or processed displacement can also be directly encoded using entropy coding.

4. Deformed mesh reconstruction module:

Since the displacement encoding stage quantization and other processes cause a certain loss of displacement, it is necessary to reconstruct the displacement at the encoding end to keep it consistent with the decoding end. After obtaining the reconstructed displacement, the deformed mesh reconstruction module subdivides the reconstructed basic mesh and obtains the reconstructed subdivided deformed mesh according to the corresponding reconstructed displacement.

5. Texture map conversion module:

The texture map conversion module first performs texture map conversion based on the input original mesh, the input original texture map, and the reconstructed subdivided deformed mesh, as shown in Figure 7. The steps of texture map conversion are as follows:

(1) Calculate the texture coordinates of each pixel on the texture map to be generated, such as the texture coordinates corresponding to pixel A(i,j) is P(u,v).

(2) Determine whether the texture coordinate is within a certain triangular face after the subdivision deformation mesh is parameterized.

(3) If the texture coordinate does not belong to any triangle, the pixel is marked as an empty pixel and can be filled with a filling algorithm.

(4) If the texture coordinate belongs to a triangle, then

a) Mark the pixel as filled.

b) Calculate the center of gravity coordinates of the texture in the current triangle surface based on the texture coordinates.

c) According to the barycentric coordinates and the corresponding triangular faces, the two-dimensional texture coordinates are mapped to three-dimensional geometric coordinates, that is, mapped to the points on the subdivided deformation grid corresponding to the texture coordinates, as shown by M(x, y, z) in the figure.

d) Find the point on the input original grid that is closest to the three-dimensional coordinate, as shown in the figure M'(x,y,z).

e) The barycentric coordinates of the three-dimensional coordinates are calculated according to the triangular face on which they are located and mapped to two dimensions to calculate their texture coordinates, namely P'(u',v').

f) Sample the input original texture map using the texture coordinates to obtain the value A'(i',j') of the corresponding pixel position.

g) Assign the value to the corresponding pixel A(i,j) on the texture map to be generated.

For empty pixels, existing filling algorithms (such as Push-Pull algorithm) may be used to fill these empty pixels.

6. Texture compression module:

The texture image compression module performs optional color space conversion and other processing on the converted texture image and then encodes it using a video encoder.

At present, the overall framework of a VDMC decoder is shown in Figure 8. For the received code stream, the decoding end first demultiplexes each part of the code stream to obtain the basic grid code stream, the displacement video code stream, and the texture map video code stream. For the basic grid code stream, the basic grid is obtained by decoding using the grid decoder corresponding to the encoding end. The displacement video code stream and the texture map video code stream are decoded by the video decoder. For the displacement part, after the video is decoded, the displacement needs to be taken out from the image through the displacement decoding module, and dequantization, inverse transformation and other steps are performed, and then it is applied to the subdivided basic grid to obtain the deformed grid reconstructed by the decoding end. After decoding, the texture map is the texture map corresponding to the reconstructed deformed grid. The subsequent application or rendering module processes the reconstructed deformed grid and the decoded texture map as input.

Next, the main modules in the decoder are described.

1. Basic grid decoding module:

The basic grid decoding module performs corresponding decoding according to the grid coding format of the input code stream. If it is intra-frame mode, the corresponding static grid decoder indicated by the auxiliary information is used for decoding; if it is inter-frame mode, the corresponding motion vector is decoded and the corresponding basic grid is reconstructed according to the reference image in the grid buffer; if it is skip mode, the corresponding reference image in the grid buffer is directly used as the basic grid of the current image.

2. Displacement decoding module and displacement reconstruction module.

The displacement video code stream is decoded by the displacement decoding module, and at the same time, the corresponding level of decoding is performed according to the LoD level required by the decoding end. If the encoder compresses the displacement by video encoding, the decoder decodes it by the corresponding video decoder, and restores it from the two-dimensional image in the corresponding order according to the arrangement scheme through the displacement reconstruction module. Then perform inverse transformation, inverse quantization and other operations on it to restore the displacement consistent with the encoder. If entropy coding is used, it can be directly entropy decoded.

3. Deformed mesh reconstruction module:

After the basic mesh and the displacement decoding and reconstruction of the corresponding level are completed, the subdivided deformed mesh is reconstructed based on these two parts. If the auxiliary information indicates that the displacement encoding process uses the reference image as a reference, that is, the displacement of the vertices in the area matching the reference image in the decoded displacement is the residual of the displacement of the current frame vertex relative to the reference image vertex, then first restore the displacement of the vertices in the current image matching area through the vertices and residuals of the subdivided deformed mesh reconstructed from the reference image, and then perform subsequent reconstruction steps. That is, the reconstructed basic mesh is subdivided, consistent with the subdivision method of the encoding end. The corresponding displacement is added to each vertex of the subdivided mesh. For the texture coordinates of the reconstructed mesh, it is obtained by midpoint subdivision interpolation consistent with the encoding end.

However, in the above encoding and decoding scheme, mesh parameterization is used to generate texture coordinates for the simplified mesh, and since the geometric coordinates of the parameterized mesh undergo subdivision deformation and quantization during the encoding and decoding process, the vertex geometric coordinates will change as shown in Figure 9. Therefore, the generated texture coordinates will not match the reconstructed deformed mesh.

Moreover, in the subsequent process of reconstructing the deformed mesh, since the vertex geometric coordinates need to be displaced after subdivision, there is a certain degree of deformation, and its texture coordinates are only obtained through fixed subdivision interpolation, which may not match the deformed geometric coordinates. Figure 10 shows the changes in the mesh shapes of each level, taking subdivision three times as an example.

In summary, the texture coordinates generated by the current mesh parameterization do not match the reconstructed deformed mesh, which reduces the decoding accuracy and thus the decoding performance. Therefore, the texture coordinates of the reconstructed deformed mesh need to be adjusted to match the deformed geometric coordinates.

The embodiment of the present application provides a coding method that can measure the parametric distortion of the texture coordinates of the vertices in the reconstructed deformed mesh, and reduce the parametric distortion through a compensation algorithm. The parametric quality compensation algorithm proposed in the embodiment of the present application can be applied to a video-based dynamic mesh coding framework to solve the parametric distortion caused by the change in the position of the mesh geometric vertices, thereby improving the quality of the reconstructed deformed mesh, and further improving the decoding accuracy and decoding performance.

The embodiment of the present application also provides a grid architecture of a codec system including a decoding method and an encoding method, and FIG11 is a schematic diagram of a grid architecture of a codec provided by the embodiment of the present application. As shown in FIG11, the grid architecture includes one or more electronic devices 13 to 1N and a communication grid 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication grid 01. During the implementation process, the electronic device can be various types of devices with codec functions, for example, the electronic device can include a mobile phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensor device, a server, etc., which are not specifically limited here.

Here, the decoder or encoder described in the embodiment of the present application may be the above-mentioned electronic device.

The embodiments of the present application are described in detail below with reference to the accompanying drawings.

Referring to Figure 12, it shows a schematic flow chart of a decoding method provided by an embodiment of the present application. As shown in Figure 12, the method may include:

S101, parsing a bit stream to determine a first basic grid and texture coordinate information of the first basic grid.

In one case of S101, the bitstream includes a first basic mesh bitstream. The first basic mesh bitstream is determined by encoding the first basic mesh including texture coordinate information. The decoder can directly parse the first basic mesh including texture coordinate information from the bitstream, thereby determining the first basic mesh and the texture coordinate information of the first basic mesh. Exemplarily, the decoder demultiplexes the bitstream to determine the bitstream of the basic mesh, and the decoder parses the bitstream of the basic mesh to determine the first basic mesh including texture coordinate information.

In some embodiments, the basic grid decoding module performs corresponding decoding according to the grid coding format of the input code stream. If it is an intra-frame mode, the corresponding static grid decoder indicated by the auxiliary information is used for decoding; if it is an inter-frame mode, the corresponding motion vector is decoded, and the first basic grid is reconstructed according to the reference image in the grid buffer; if it is a skip mode, the corresponding reference image in the grid buffer is directly used as the first basic grid.

In some embodiments, the displacement code stream is decoded by a displacement decoder, and at the same time, the corresponding level of decoding is performed according to the LoD level required by the decoding end. If the encoding end compresses the displacement by video encoding, the decoding end decodes it by a corresponding video decoder and restores it from the two-dimensional image in the corresponding order according to the arrangement scheme. Then, inverse transformation, inverse quantization and other operations are performed on it to restore the displacement consistent with the encoding end. If entropy encoding is adopted, it can be directly entropy decoded.

For another case of S101, the code stream includes a second basic grid code stream and a texture coordinate code stream. Among them, the second basic network code stream is determined by encoding the second basic grid that does not contain texture coordinate information. The decoder demultiplexes the code stream to determine the second basic grid code stream and the texture coordinate code stream. The decoder parses the second basic grid code stream to determine the second basic grid. Here, the process of parsing the second basic grid is consistent with the above-mentioned process description of parsing the first basic grid, and will not be repeated here. The decoder parses the texture coordinate code stream to determine the texture coordinate information; based on the texture coordinate information, the second basic grid is reconstructed to determine the first basic grid and the texture coordinate information corresponding to the first basic grid.

In some embodiments, the second base mesh is reconstructed according to the texture coordinate information to determine the base mesh including the texture coordinate information; the base mesh including the texture coordinate information is cleaned to determine the first base mesh and the texture coordinate information of the first base mesh.

S102: Perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh.

S103 . Determine texture coordinate information of a reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh.

In S102, the decoder also determines the displacement information corresponding to the first basic mesh by parsing the bitstream. The decoder performs mesh subdivision and deformation based on the first basic mesh to determine the basic subdivided mesh and the reconstructed deformed mesh; wherein the reconstructed deformed mesh includes three-dimensional coordinate information, that is, the geometric coordinates of the vertices in the reconstructed deformed mesh. Based on the texture coordinate information of the first basic mesh and the basic subdivided mesh, the initial texture coordinate information of the reconstructed deformed mesh is determined; based on the initial texture coordinate information of the reconstructed deformed mesh, the texture coordinate information of the reconstructed deformed mesh is determined.

The basic subdivided mesh is determined by subdividing the first basic mesh, and the reconstructed deformed mesh is determined by superimposing displacement information on vertices in the basic subdivided mesh using displacement information parsed from the bitstream. The decoder subdivides and interpolates the texture coordinate information of the first basic mesh according to the basic subdivided mesh corresponding to the first basic mesh, as the initial texture coordinate information of the reconstructed deformed mesh. The texture coordinate information of the reconstructed deformed mesh includes the texture coordinates of each vertex in the reconstructed deformed mesh.

In some embodiments, after the first basic mesh and the displacement decoding and reconstruction of the corresponding level are completed, the subdivided deformed mesh is reconstructed based on the two parts, that is, the deformed mesh is reconstructed. If the auxiliary information indicates that the displacement encoding process uses a reference image as a reference, that is, the displacement of the vertices in the area matching the reference image in the decoded displacement is the residual of the displacement of the current image vertex relative to the reference image vertex, then firstly, the displacement of the vertices in the current image matching area is restored through the vertices and residuals of the subdivided deformed mesh reconstructed from the reference image, and then subsequent reconstruction steps are performed. That is, the reconstructed basic mesh is subdivided, which is consistent with the subdivision method of the encoding end. The corresponding displacement is added to each vertex of the subdivided mesh. For the texture coordinates of the reconstructed mesh, it is obtained by midpoint subdivision interpolation consistent with the encoding end.

It can be seen that in the reconstruction of the deformed mesh of the corresponding level, the geometric position of the vertices in the reconstructed deformed mesh is obtained by midpoint subdivision interpolation and adding the corresponding displacement, while the corresponding texture coordinates are only midpoint subdivision interpolation, which has a certain deviation from the geometric position after adding the displacement. Therefore, it is necessary to compensate its texture coordinates to match the reconstructed deformed mesh after geometric deformation.

In some embodiments, based on the decoder framework shown in FIG8 , a decoder framework of an embodiment of the present application can be shown in FIG13 . After the decoding end obtains the code stream, it first demultiplexes each part of the code stream. The basic grid decoding module decodes the basic grid code stream to determine the basic grid and the texture coordinate information of the basic grid (that is, the texture coordinate information of the first basic grid and the first basic grid). Then, according to the detail level required by the decoding end, the required displacement level is decoded and reconstructed, and then the reconstructed displacement is applied to the subdivided basic grid to reconstruct the deformed grid to obtain the reconstructed deformed grid of the detail level required by the decoding end. The reconstructed deformed grids of different detail levels can be described as the reconstructed deformed grid of the L1 level, the reconstructed deformed grid of the L2 level, and the reconstructed deformed grid of the L3 level in FIG12 . Texture coordinate compensation is performed on the texture coordinates of the reconstructed deformed grid of one level, and the texture coordinates of the vertices in the reconstructed deformed grid of the current level are adjusted to reduce its parameterized distortion, and finally the deformed grid output by the decoding end is obtained. The texture map corresponding to the grid is obtained by video decoding the texture map code stream.

In some embodiments, the texture coordinates of the reconstructed deformed mesh can be updated by a variety of compensation algorithms, and the reconstructed deformed mesh after geometric deformation. Exemplarily, the texture coordinates of the reconstructed deformed mesh can be updated directly according to the displacement reconstructed by decoding to compensate for the deviation between the geometric coordinates of the reconstructed deformed mesh with the displacement added due to only midpoint subdivision interpolation of the texture coordinates. Alternatively, since the three-dimensional space to the two-dimensional space is not equidistant, the parameterization process will cause distortion. Distortion can be measured in many ways, including the preservation of angles or areas, or how much parameter distance is stretched or shrunk on the surface, etc. According to the measurement method of distortion, a corresponding compensation algorithm can be designed to compensate or update the texture coordinates of the reconstructed deformed mesh. The specific selection is based on the actual situation, and the embodiments of the present application are not limited.

In some embodiments, based on the decoder framework shown in FIG. 7 , another decoder framework of the embodiment of the present application can be shown in FIG. 14 . After the decoding end obtains the code stream, it first demultiplexes each part of the code stream. The basic grid decoding module decodes the basic grid code stream. The basic grid decoding module decodes the basic grid code stream, determines the second basic grid, and then reconstructs the texture coordinates of the second basic grid according to whether the code stream contains the texture coordinate code stream (that is, the texture coordinate reconstruction is optional), reconstructs the second basic grid containing the texture coordinates, and performs grid cleaning on the second basic grid containing the texture coordinates to obtain the first basic grid, and the displacement level of the first basic grid corresponds to the L0 level. According to the level of detail required by the decoder, the corresponding displacement level is decoded and reconstructed, and then the reconstructed displacement is applied to the subdivided first basic grid to obtain the deformed grid of the detail level required by the decoder, that is, the reconstructed deformed grid. Then, the texture coordinates of the vertices in the reconstructed deformed grid are adjusted, and its parameterized distortion is reduced according to the preset compensation algorithm, and finally the deformed grid output by the decoding end is obtained. The texture map corresponding to the grid is obtained by video decoding the texture map code stream. In this way, the texture coordinates sent by the encoder can be used to further improve the quality of the reconstructed deformed mesh and improve the decoding accuracy and decoding performance.

It can be understood that in the embodiment of the present application, by updating the texture coordinates of the first basic mesh determined by the parsed code stream, the distortion of the texture coordinates of the vertices is reduced, the accuracy of the texture coordinates and the matching degree with the reconstructed deformed mesh are improved, thereby improving the quality of the reconstructed deformation and improving the decoding performance.

In some embodiments, in the above decoding method provided in the embodiment of the present application, the process of determining the texture coordinate information of the reconstructed mesh may be as shown in FIG. 15, including:

S201. Determine initial texture coordinate information of a reconstructed deformed mesh based on texture coordinate information of a first basic mesh and a basic subdivided mesh.

S202: Determine a first distortion measure and/or a second distortion measure based on initial texture coordinate information of the reconstructed deformed mesh.

In some embodiments, based on the initial texture coordinate information of the reconstructed mesh and the three-dimensional coordinate information of the reconstructed mesh, the texture coordinates and three-dimensional coordinates of the triangles corresponding to the vertices in the reconstructed deformed mesh are determined; based on the texture coordinates and three-dimensional coordinates of the triangles corresponding to the current vertex in the reconstructed deformed mesh, the first distortion measure and/or the second distortion measure is determined.

The embodiment of the present application measures the distortion of a single vertex by two measures: average stretching of the local distance of the surface and stretching in the worst case. The first distortion measure represents the average stretching of the local distance of the surface of the triangular face from the three-dimensional coordinate space to the texture coordinate space; the second distortion measure represents the worst case stretching of the triangular face from the three-dimensional coordinate space to the texture coordinate space.

In some embodiments, for a current vertex among the at least one vertex, determining the first distortion measure and/or the second distortion measure according to the texture coordinates and the three-dimensional coordinates of the triangle corresponding to the current vertex in the reconstructed deformed mesh includes:

Determine a first length parameter and a second length parameter based on texture coordinates and three-dimensional coordinates of vertices in the triangle patch; determine a first distortion measure based on the first length parameter and the second length parameter; and/or determine a second distortion measure based on the second length parameter.

The first length parameter represents the minimum length of the unit length vector mapped from the texture coordinate space to the three-dimensional coordinate space; the second length parameter represents the maximum length of the unit length vector mapped from the texture coordinate space to the three-dimensional coordinate space;

In some embodiments, the first length parameter and the second length parameter may be determined by:

Determine a first partial derivative and a second partial derivative according to an affine mapping between texture coordinates of vertices in the triangular patch and three-dimensional coordinates;

Determine the partial derivative matrix corresponding to the vertices in the triangular patch according to the first partial derivative and the second partial derivative;

Determine a first length parameter according to a first singular value corresponding to the partial derivative matrix;

A second length parameter is determined according to a second singular value corresponding to the partial derivative matrix; the first singular value is greater than the second singular value.

Exemplarily, the two-dimensional texture coordinates of the three vertices of the triangle patch T corresponding to the current vertex are p ₁ , p ₂ , p ₃ , respectively, where p _i =(s _i ,t _i ), i={1,2,3}. The three-dimensional geometric coordinates of the triangle patch T are expressed as q ₁ ,q ₂ ,q ₃ . The calculation process of the texture coordinate of any point p on the triangle patch T in the texture coordinate space (i.e., two-dimensional space) to the point affine mapping relationship S(p)=S(s,t)=q on the triangle patch in three-dimensional space can be shown as formula (1), as follows:
S(p)＝(<p,p ₂ ,p ₃ >q ₁ +<p,p ₃ ,p ₁ >q ₂ +<p,p ₁ ,p ₂ >q ₃ )/<p ₁ ,p ₂ ,p ₃ > (1)

In formula (1), <a, b, c> represents the area of the triangle formed by the three vertices a, b, and c; for example, <p, p ₂ , p ₃ > represents the area of the triangle formed by the three vertices p p, p ₂ , and p ₃ .

Since the mapping is affine, its partial derivative is a constant at the two-dimensional coordinates (s, t) of point p. According to the above affine mapping, the first partial derivative can be determined by calculating the partial derivative of S(p) at the coordinate s, as shown in formula (2), as follows.

According to the above affine mapping, by calculating the partial derivative of S(p) at coordinate t, the second partial derivative can be determined as shown in formula (3), as follows.

In formula (2), S _s is the first partial derivative, and in formula (3), S _t is the second partial derivative. In formula (2) and formula (3),
A＝<p ₁ , p ₂ , p ₃ >=((s ₂ -s ₁ )(t ₃ -t ₁ )-(s ₃ -s ₁ )(t ₂ -t ₁ ))/2.

According to the first partial derivative S _s and the second partial derivative St _t , the partial derivative matrix corresponding to the vertex in the triangular patch is determined, such as determining the Jacobian matrix [S _s , _St ], and the larger singular value γ _max and the smaller singular value γ _min of the Jacobian matrix are determined by formula (4) and formula (5), which are used as the first singular value and the second singular value, respectively, as follows:

In formula (4) and formula (5), a = S _s ·S _s , b = S _s ·S _t , c = S _t ·S _t . γ _max is the first singular value, and γ _max is used as the first length parameter to represent the maximum length obtained when a unit length vector is mapped from a two-dimensional texture domain to a three-dimensional surface, that is, the maximum local "stretch". γ _min is the second singular value, and γ _min is used as the second length parameter to represent the minimum length obtained when a unit length vector is mapped from a two-dimensional texture domain to a three-dimensional surface, that is, the minimum local "stretch".

In some embodiments, the first distortion measure L ² (T) may be determined according to the first length parameter γ _max and the second length parameter γ _min by formula (6), as follows.

In some embodiments, the second distortion measure L ^∞ (T) may be determined according to the second length parameter γ _min by formula (7), as follows:
L ^∞ (T) = γ _max (7)

In some embodiments, according to formula (4), formula (5) and formula (6), we can get: That is, the first distortion measure is determined according to a first dot product a=S _s ·S _s corresponding to the first partial derivative S _s and a second dot product c=S _t ·S _t corresponding to the second partial derivative _S t.

S203: Determine texture coordinate information of the reconstructed deformed mesh according to the first distortion measure and/or the second distortion measure.

In the embodiment of the present application, the first distortion measure L ² (T) and/or the second distortion measure L ^∞ (T) can measure the parameterized distortion of the current vertex. For the current vertex, its corresponding texture coordinates need to be adjusted to minimize the parameterized distortion of the current vertex.

In some embodiments, updating the texture coordinates of the current vertex according to the first distortion measure and/or the second distortion measure includes:

Determining a texture coordinate error corresponding to a current vertex in the reconstructed deformed mesh according to the first distortion measure and/or the second distortion measure;

The texture coordinates of the current vertex are updated by minimizing the texture coordinate error, thereby determining the texture coordinate information of the reconstructed deformed mesh.

In some embodiments, the triangle corresponding to the current vertex includes: at least one triangle adjacent to the current vertex; the texture coordinate error includes: a first texture coordinate error and/or a second texture coordinate error;

determining a first texture coordinate error according to a sum of first distortion measures corresponding to at least one triangular face;

and/or,

A second texture coordinate error is determined according to a maximum second distortion measure corresponding to at least one triangular facet.

Exemplarily, the first texture coordinate error may be determined by formula (8), as follows:

In formula (8), f represents the triangle face adjacent to the current vertex vt, that is, f∈δ _vt . It can be seen that the first distortion measure L ² (f) is calculated for each triangle face in at least one triangle face adjacent to the current vertex vt, and the sum of at least one first distortion measure corresponding to at least one triangle face is calculated according to A first texture coordinate error L ² (vt) is determined.

Exemplarily, the second texture coordinate error may be determined by formula (9), as follows:

In formula (9), according to the maximum second distortion measure corresponding to at least one triangle patch A second texture coordinate error ^L∞ (vt) is determined.

In some embodiments, the process of updating the texture coordinates of the current vertex by minimizing the texture coordinate error includes:

Update the texture coordinates of the current vertex by minimizing the first texture coordinate error;

or,

Update the texture coordinate of the current vertex by minimizing the second texture coordinate error;

or,

Update the texture coordinates of the current vertex by minimizing the first texture coordinate error and the second texture coordinate error respectively.

It should be noted that the process of minimizing the first texture coordinate error and the second texture coordinate error includes: a process of updating the texture coordinates of the current vertex by minimizing the first texture coordinate error, and a process of updating the texture coordinates of the current vertex by minimizing the second texture coordinate error. That is to say, the first texture coordinate error and the second texture coordinate error are optimized separately in the embodiment of the present application. The embodiment of the present application does not limit the optimization order corresponding to the first texture coordinate error and the second texture coordinate error. Moreover, when the texture coordinates are updated by the subsequent texture coordinate error, the texture coordinates of the vertex after the update by the previous texture coordinate error are updated.

It can be understood that in the embodiments of the present application, the distortion of the texture coordinates of the vertices in the reconstructed deformed mesh is measured by two measures: the average stretching of the surface local distance of the vertex-adjacent triangles from the three-dimensional coordinate space to the texture coordinate space and/or the worst-case stretching of the triangles from the three-dimensional coordinate space to the texture coordinate space. The texture coordinates of the vertices are adjusted according to the distortion, the texture coordinates of the vertices are updated, and the accuracy of the texture coordinates and the matching degree with the reconstructed deformed mesh are improved, thereby improving the quality of the reconstructed deformed mesh and improving the decoding performance.

Next, based on FIG. 16 , the process of updating the texture coordinates of the current vertex by minimizing the texture coordinate error is described.

It should be noted that, in the case of updating the texture coordinates of the current vertex by minimizing the first texture coordinate error, the texture coordinate error in the following process includes the first texture coordinate error. In the case of updating the texture coordinates of the current vertex by minimizing the second texture coordinate error, the texture coordinate error in the following process includes the second texture coordinate error. In the case of updating the texture coordinates of the current vertex by minimizing the first texture coordinate error and the second texture coordinate error respectively, when the texture coordinates are updated by minimizing the first texture coordinate error, the texture coordinate error in the following process includes the first texture coordinate error; when the texture coordinates are updated by minimizing the second texture coordinate error, the texture coordinate error in the following process includes the second texture coordinate error.

S301. Determine the initial center point and search range based on the current vertex.

In S301, the decoder may determine the current vertex as the initial center position point; or determine at least one candidate position point corresponding to the current vertex; determine the initial center position point from at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point. The decoder determines the search range based on the initial center position point.

In some embodiments, at least one candidate position point may include: a point at the average position of the adjacent points of the current vertex, and/or a midpoint of the adjacent edge of the current vertex. Exemplarily, at least one candidate position point corresponding to the current vertex may be as shown in FIG. 17 , where vt is the current vertex, and vt1′ to vt7′ are at least one candidate position point corresponding to the current vertex.

In some embodiments, the process of determining the initial center position point from at least one candidate position point by determining at least one candidate texture coordinate error corresponding to at least one candidate position point may include: determining at least one candidate texture coordinate error corresponding to at least one candidate position point; and determining an initial center position point according to a minimum value of at least one candidate texture coordinate error and a texture coordinate error corresponding to a current vertex.

Exemplarily, the texture coordinate error corresponding to each candidate position point in FIG. 17 is determined by formula (8) and/or formula (9) as the candidate texture coordinate error corresponding to each candidate position point. By comparing at least one candidate texture coordinate error, the minimum candidate texture coordinate error is determined. If the minimum candidate texture coordinate error is smaller than the texture coordinate error corresponding to the current vertex, the candidate position point corresponding to the minimum candidate texture coordinate error is used as the initial center position point. If the minimum candidate texture coordinate error is larger than the texture coordinate error corresponding to the current vertex, the current vertex is used as the initial center position point.

In some embodiments, the decoder determines the search range with the initial center position point as the center and the minimum value of at least one distance from the initial center position point to the third edge of at least one adjacent face as the radius. Exemplarily, the search range can be as shown in the dotted circle of FIG18, with the initial center position point as the center, constructing an adjacent circle as the search range, and the radius of the circle is the minimum value of the distance from the vertex to the third edge of its adjacent face.

S302: Perform at least one search based on the initial center position point and the search range to determine at least one initial target position point.

In S302, the decoder uses the initial center position point as the starting point and performs at least one search within the search range to search for a position point with smaller candidate texture coordinates to update the texture coordinates. Here, a variety of search algorithms can be used, and the specific selection is based on actual conditions, which is not limited in the embodiments of the present application.

For example, based on formula (8) and formula (9), the objective function for minimizing the texture coordinate error can be determined as shown in formula (10), as follows:

Where N is the search range and vt′ is the position point found each time.

Exemplarily, taking binary search as an example, the process of S202 may include:

Determine the search direction for each search; based on the initial center position point, perform a binary search within the search range according to the search direction of each search to determine the initial target position point corresponding to each search until at least one search is completed and at least one initial target position point is determined.

For binary search, in each search of at least one search, the initial center position point is used as the starting point of each search, and the end point of each search is determined within the search range according to the search direction of each search; when the candidate texture coordinate error corresponding to the starting point is less than the candidate texture coordinate error corresponding to the end point, the end point is updated according to the midpoint between the starting point and the end point; when the candidate texture coordinate error corresponding to the starting point is greater than the corresponding candidate texture coordinate error of the end point, the starting point is updated according to the midpoint between the starting point and the end point; the above process is used for at least one update until it is determined that the distance between the updated starting point and the updated end point is less than a preset distance threshold, and the initial target position point corresponding to each search is determined according to the starting point and/or the end point.

In some embodiments, when the distance between the updated starting point and the updated ending point is less than a preset distance threshold at least once in each search, it is determined that the distance between the updated starting point and the updated ending point is less than the preset distance threshold.

In some embodiments, when it is determined that the distance between the updated starting point and the updated ending point is less than a preset distance threshold, the starting point obtained by the current update can be determined as the initial target position point; or the ending point obtained by the current update can be determined as the initial target position point; or the midpoint between the starting point obtained by the current update and the ending point obtained by the current update can be determined as the initial target position point; or the point with the smallest error in the candidate texture coordinates between the starting point obtained by the current update and the ending point obtained by the current update can be determined as the target position point. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited thereto.

Exemplarily, within the search range, a direction is randomly selected as the search direction. The direction is selected by random number generation, and the randomly selected direction is more likely to find the optimal solution; the optimal value is gradually approached in the selected direction by bisection. That is, the errors corresponding to the two endpoints are calculated, the center of the circle is recorded as the starting point, and the other end is recorded as the end point. If the error of the starting point is less than the end point, the midpoint of the two end points is used as the end point; if the error of the starting point is greater than the end point, the midpoint of the two end points is used as the starting point, and then the error of the two end points is calculated for comparison. Iterate in the above manner until the distance between the vertices selected twice in a row is less than the preset threshold, complete a search, and determine the initial target position point corresponding to the search. In order to increase the stability of the search, directions are randomly selected multiple times, and the above steps are repeated, that is, at least one search is performed to find the optimal local solution, and the target position point is determined from at least one initial target position point.

S303: Determine a target position point according to a minimum candidate texture coordinate error corresponding to at least one initial target position point.

S304: Update the texture coordinates of the current vertex according to the texture coordinates of the target position point.

In S303-S304, the initial target position point with the smallest error in the candidate texture coordinates among at least one initial target position point may be used as the determined target position point. It is understandable that the target position point is the position point with the smallest distortion within the search range of the current vertex. The decoder updates the texture coordinates of the current vertex according to the texture coordinates of the target position point. Exemplarily, the texture coordinates of the current vertex are updated to the texture coordinates of the target position point.

It can be understood that by searching for the target position point corresponding to the current vertex within the local range (search range), the texture coordinates of the current vertex are updated according to the texture coordinates of the target position point with less distortion, thereby improving the matching degree between the texture coordinates of the current vertex and the current vertex, thereby improving the quality of reconstructing the deformed mesh, and further improving the decoding accuracy and decoding performance.

In some embodiments, for at least one vertex to be optimized in the reconstructed deformed mesh, at least one texture coordinate error corresponding to the at least one vertex is determined; at least one vertex is organized into a sequence to be optimized according to the at least one texture coordinate error; a vertex corresponding to a preset sequence position in the sequence to be optimized is determined as the current vertex; and the preset sequence position corresponds to a vertex with the largest texture coordinate error among the at least one vertex. Here, the at least one vertex may include some or all of the vertices in the reconstructed deformed mesh.

That is, the current vertex is the vertex with the largest texture coordinate error in the sequence to be optimized. In this way, when the target position point corresponding to the current vertex is determined, the texture coordinate error corresponding to the current vertex in at least one vertex can be updated according to the target position point, and then the order of the vertices in the sequence to be optimized is updated; the texture coordinates of the vertices at the updated preset sequence positions are updated until the preset iteration target is reached, and the texture coordinate update of the reconstructed deformed mesh is completed.

Exemplarily, for the process of selecting the current vertex for updating the texture coordinates, a preset data structure, such as a heap data structure, can be used to sort at least one vertex according to its texture coordinate error to obtain a sequence to be optimized of the heap data structure. Exemplarily, vertices with poor quality are placed at the top of the heap, while vertices with better quality are located at a lower position. In each iteration, the vertex with the largest current texture coordinate error is taken from the top of the heap for optimization. By adjusting the texture coordinates of the vertex, its texture coordinate error is reduced as much as possible. Then, the texture coordinate errors of other vertices adjacent to the vertex are updated, and they are put back into the sequence to be optimized of the heap structure. It can be seen that with each optimization iteration, the vertex at the top of the heap will change, so it is necessary to update the sequence to be optimized of the heap structure to reflect the latest error sorting. This ensures that in subsequent iterations, the vertex with the worst current quality can still be selected for adjustment. Through the above strategy, the entire optimization process is closer to the goal of global optimization. Vertices with poor quality are adjusted more frequently in iterations, thereby driving the overall parameterization to evolve in a better direction. Thereby improving decoding efficiency and decoding accuracy.

In some embodiments, when the number of updates of the sequence to be optimized reaches a preset update number threshold, and/or the texture coordinate error of the vertex at the preset sequence position is less than a preset error threshold, it is determined that the preset iteration target is reached, and the texture coordinate update of at least one vertex to be optimized in the reconstructed deformed mesh is completed.

In some embodiments, based on FIG. 13 and FIG. 14 , it can be seen that the texture coordinates of the vertices in each layer of the reconstructed deformed mesh in at least one layer of the reconstructed deformed mesh can be updated. That is, the reconstructed deformed mesh processed each time includes: the i-th layer of the reconstructed deformed mesh corresponding to the base mesh; i is greater than 1. When the decoder completes the update of the texture coordinates of the i-th layer of the reconstructed deformed mesh, the texture coordinates of the vertices in the i+1-th layer of the reconstructed deformed mesh can be interpolated based on the texture coordinates of the vertices in the i-th layer of the reconstructed deformed mesh; i+1 is less than or equal to the preset number of layers; the texture coordinates of the vertices in the i+1-th layer of the reconstructed deformed mesh are updated by determining the first distortion measure and/or the second distortion measure of the triangle facets corresponding to the vertices in the i+1-th layer of the reconstructed deformed mesh. The texture coordinates of the vertices in each layer of the reconstructed deformed mesh are adjusted layer by layer in the above process until the texture coordinate update of at least one layer of the reconstructed deformed mesh is completed.

Exemplarily, taking Figure 12 as an example, after decoding to obtain the L1 level and performing texture coordinate compensation, if the decoder needs to further improve the detail level such as obtaining the L2 level, the interpolated position of the midpoint of the L1 level texture coordinate is used as the starting point, and then the texture coordinates of the L2 level are updated, and so on, until the detail level required by the decoding end is reconstructed.

Referring to Figure 19, it shows a schematic flow chart of a coding method provided in an embodiment of the present application. As shown in Figure 19, the method may include:

S401, determining a basic grid corresponding to the current image.

S402: Determine texture coordinate information based on the basic mesh.

S403: Encode the basic grid and texture coordinate information, and write the obtained encoding bits into the bit stream.

In some embodiments, an encoder framework corresponding to the above decoding method can be as described in the current VDMC encoding framework in FIG3. That is, the input mesh is subjected to the basic mesh generation step to obtain the basic mesh and the corresponding subdivided deformed mesh, and the basic mesh is compressed and reconstructed by the basic mesh compression module to obtain the reconstructed basic mesh. Then, according to the deformed mesh obtained in the basic mesh generation stage, the displacements between the corresponding vertices of the subdivided basic mesh and the corresponding subdivided deformed mesh are obtained, and the displacements are processed by coordinate system conversion, wavelet transformation, quantization, etc., and then arranged in the image and encoded by a video encoder or directly entropy encoded. Subsequently, the encoded displacements are reconstructed to obtain a reconstructed subdivided deformed mesh, and the mesh, the original input mesh and the texture map are sent as input to the texture map conversion module to obtain the texture map corresponding to the currently reconstructed deformed mesh, and the texture map is subjected to optional color space conversion and other processing and then encoded by a video encoder.

The texture coordinate information is determined by mesh parameterization based on the base mesh. The encoder can encode the base mesh containing the texture coordinate information and write the obtained encoding bits into the bitstream. The bitstream here corresponds to the first basic bitstream of the decoder.

In this way, the decoding method in the above embodiment is executed through the decoder to compensate the texture coordinates of the reconstructed deformed mesh, and the matching degree of the texture coordinates and the reconstructed deformed mesh, thereby improving the decoding accuracy and decoding performance.

In some embodiments, unlike the above-mentioned method of merging the texture coordinate information with the base grid for encoding, since the base grid encoding is a compressed encoding, in order to reduce the distortion of the texture coordinate information, the texture coordinate information and the base grid can be encoded separately, and the bitstream corresponding to the base grid that does not contain the texture coordinate information (the second base grid bitstream corresponding to the decoder) and the bitstream corresponding to the texture coordinate information are determined separately.

In one embodiment, the texture coordinate information can be determined in the process of generating the basic mesh by the basic mesh generation module, and the texture coordinate information includes: texture coordinate information corresponding to the basic mesh. The encoder can perform mesh parameterization based on the basic mesh to determine the texture coordinate information corresponding to the basic mesh; perform compression encoding on the basic mesh to determine the coding bits corresponding to the basic mesh; encode the texture coordinate information to determine the coding bits corresponding to the texture coordinate information; and write the coding bits corresponding to the basic mesh and the coding bits corresponding to the texture coordinate information into the bitstream respectively. In this way, the second basic mesh parsed from the basic bitstream can be reconstructed in the decoder by parsing the texture coordinate information in the bitstream to determine the first basic mesh, so as to improve the accuracy of the texture coordinates in the first basic mesh, thereby improving the coding and decoding accuracy and the coding and decoding performance.

Exemplarily, as shown in FIG20, the input mesh first passes through the basic mesh generation step to obtain a basic mesh, and the basic mesh is compressed and reconstructed by the basic mesh compression module to obtain a reconstructed basic mesh. The texture coordinates obtained in the basic mesh generation stage can be restored according to the texture coordinate information of the current image, so this information can be passed to the decoding end to avoid directly encoding the texture coordinates. After the texture coordinates are regenerated for the reconstructed basic mesh, the mesh is cleaned up, that is, duplicate points and degenerate surfaces are removed. Then, after the subdivision deformation step, the displacement between the corresponding vertices of the subdivided basic mesh and the corresponding subdivided deformed mesh is obtained, and the subsequent processing process is the same as the corresponding encoding process in FIG3.

In another embodiment, the texture coordinate information can be determined based on the reconstructed base grid, and the texture coordinate information includes: texture coordinate information corresponding to the reconstructed base grid. The encoder can reconstruct based on the coding bits of the base grid to determine the reconstructed base grid; perform mesh parameterization on the reconstructed base grid to determine the texture coordinate information; perform compression coding on the base grid to determine the coding bits corresponding to the base grid; encode the texture coordinate information to determine the coding bits corresponding to the texture coordinate information; and write the coding bits corresponding to the base grid and the coding bits corresponding to the texture coordinate information into the bitstream respectively.

Exemplarily, as shown in FIG21, after the input mesh is generated by the basic mesh module, the basic mesh is directly compressed to obtain a basic mesh code stream and a reconstructed basic mesh, and the reconstructed basic mesh is mesh parameterized to obtain a reconstructed mesh containing texture coordinates, and the texture coordinate information is also encoded as a basic mesh code stream. Then, the parameterized basic mesh is mesh cleaned to remove its duplicate vertices and degenerate faces, and a subdivision deformation operation is performed to obtain the corresponding displacement, and the subsequent processing process is the same as the corresponding encoding process in FIG3.

It can be understood that by separately encoding the texture coordinate information and sending it to the decoder, the distortion of the texture coordinate information is reduced, the encoding and decoding accuracy is further improved, and thus the encoding and decoding performance is improved.

In another embodiment of the present application, based on the same inventive concept as the above-mentioned embodiment, see Figure 22, which shows a schematic diagram of the composition structure of an encoder provided by an embodiment of the present application. As shown in Figure 22, the encoder 190 may include a grid information determination part 1901 and an encoding part 1902, wherein:

The mesh generation part 1901 is configured to determine a base mesh corresponding to the current image; and determine texture coordinate information based on the base mesh;

The encoding part 1902 is configured to encode the basic grid and the texture coordinate information, and write the obtained encoding bits into a bit stream.

In some embodiments, the texture coordinate information includes: texture coordinate information corresponding to the basic mesh; the mesh information determination part 1901 is further configured to perform mesh parameterization based on the basic mesh to determine the texture coordinate information.

In some embodiments, the texture coordinate information includes: texture coordinate information corresponding to the reconstructed basic grid; the grid information determination part 1901 reconstructs based on the coding bits of the basic grid to determine the reconstructed basic grid; and grid parameterizes the reconstructed basic grid to determine the texture coordinate information.

In some embodiments, the encoding part 1902 is further configured to encode the basic grid including the texture coordinate information, and write the obtained encoding bits into a bitstream.

In some embodiments, the encoding part 1902 is further configured to perform compression encoding on the basic grid to determine the encoding bits corresponding to the basic grid; to encode the texture coordinate information to determine the encoding bits corresponding to the texture coordinate information; and to write the encoding bits corresponding to the basic grid and the encoding bits corresponding to the texture coordinate information into the bit stream respectively.

It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

It is understandable that in the embodiments of the present application, "part" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in the present embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or grid device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 190. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, the method described in any one of the aforementioned embodiments is implemented.

Based on the composition of the encoder 190 and the computer-readable storage medium, refer to Figure 23, which shows a specific hardware structure diagram of the encoder 190 provided in an embodiment of the present application. As shown in Figure 23, the encoder 190 may include: a first communication interface 2301, a first memory 2302 and a first processor 2303; each component is coupled together through a first bus system 2304. It can be understood that the first bus system 2304 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 2304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 2304 in Figure 20. Among them,

The first communication interface 2301 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

A first memory 2302, used to store a computer program that can be run on the first processor 2303;

The first processor 2303 is configured to execute the encoding method applied to the encoder in the embodiment of the present application when running the computer program.

It can be understood that the first memory 2302 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). The first memory 2302 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

The first processor 2303 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the first processor 2303. The above-mentioned first processor 2303 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the first memory 2302, and the first processor 2303 reads the information in the first memory 2302 and completes the steps of the above method in combination with its hardware.

It is understood that the embodiments described in this application can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application or a combination thereof. For software implementation, the technology described in this application can be implemented by a module (such as a process, function, etc.) that performs the functions described in this application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

Optionally, as another embodiment, the first processor 2303 is further configured to execute the encoding method applied to the encoder described in any one of the aforementioned embodiments when running the computer program.

Based on the same inventive concept as the above-mentioned embodiment, referring to FIG. 24 , a schematic diagram of the composition structure of a decoder provided in an embodiment of the present application is shown. As shown in FIG. 24 , the decoder 240 may include the parsing part 2401 and the coordinate determination part 2403, wherein:

The parsing part 2401 is configured to parse the code stream to determine the first basic grid and the texture coordinate information of the first basic grid;

The subdivision and deformation part 2402 is configured to perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh;

The coordinate determining part 2403 is configured to determine the texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh.

In some embodiments, the parsing part 2401 is further configured to demultiplex the code stream to determine a first basic grid code stream;

The first basic grid code stream is parsed to determine the first basic grid including the texture coordinate information, thereby determining the first basic grid and the texture coordinate information of the first basic grid.

In some embodiments, the parsing part 2401 is further configured to demultiplex the code stream to determine a second basic grid code stream and a texture coordinate code stream; parse the second basic grid code stream to determine a second basic grid; parse the texture coordinate code stream to determine texture coordinate information; reconstruct the second basic grid based on the texture coordinate information to determine the first basic grid and the texture coordinate information of the first basic grid.

In some embodiments, the decoder 240 also includes a reconstruction part, which is configured to reconstruct the second basic grid according to the texture coordinate information to determine the basic grid containing the texture coordinate information; perform grid cleaning on the basic grid containing the texture coordinate information to determine the first basic grid and the texture coordinate information of the first basic grid.

In some embodiments, the subdivision deformation part 2402 is further configured to perform grid subdivision based on the first basic grid to determine a basic subdivision grid; and determine the reconstructed deformed grid based on the basic subdivision grid and displacement information parsed from the bitstream.

In some embodiments, the coordinate determination part 2403 is also configured to determine the initial texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the basic subdivision mesh; determine a first distortion measure and/or a second distortion measure based on the initial texture coordinate information of the reconstructed deformed mesh; the first distortion measure represents the average stretching of the surface local distance of the triangular facets in the reconstructed deformed mesh from the three-dimensional coordinate space to the texture coordinate space; the second distortion measure represents the worst-case stretching of the triangular facets in the reconstructed deformed mesh from the three-dimensional coordinate space to the texture coordinate space; determine the texture coordinate information of the reconstructed deformed mesh according to the first distortion measure and/or the second distortion measure.

In some embodiments, the coordinate determination part 2403 is also configured to determine the texture coordinates and three-dimensional coordinates of the triangular facets corresponding to the vertices in the reconstructed deformed mesh based on the initial texture coordinate information of the reconstructed mesh and the three-dimensional coordinate information of the reconstructed mesh; determine the first length parameter and the second length parameter according to the texture coordinates and the three-dimensional coordinates of the triangular facets; the first length parameter represents the minimum length corresponding to the mapping of a unit length vector from the texture coordinate space to the three-dimensional coordinate space; the second length parameter represents the maximum length corresponding to the mapping of a unit length vector from the texture coordinate space to the three-dimensional coordinate space; determine the first distortion measure according to the first length parameter and the second length parameter; and/or determine the second distortion measure according to the second length parameter.

In some embodiments, the coordinate determination part 2403 is also configured to determine the first partial derivative and the second partial derivative according to the affine mapping between the texture coordinates of the triangular patch and the three-dimensional coordinates; determine the partial derivative matrix corresponding to the vertices in the triangular patch according to the first partial derivative and the second partial derivative; determine the first length parameter according to the first singular value corresponding to the partial derivative matrix; determine the second length parameter according to the second singular value corresponding to the partial derivative matrix; the first singular value is greater than the second singular value.

In some embodiments, the coordinate determining part 2403 is further configured to determine the first distortion measure based on a first dot product corresponding to the first partial derivative and a second dot product corresponding to the second partial derivative.

In some embodiments, the coordinate determination part 2403 is also configured to determine the texture coordinate error corresponding to the current vertex in the reconstructed deformed mesh based on the first distortion measure and/or the second distortion measure; and update the texture coordinates of the current vertex by minimizing the texture coordinate error, thereby determining the texture coordinate information of the reconstructed deformed mesh.

In some embodiments, the triangular face corresponding to the current vertex includes: at least one triangular face adjacent to the current vertex; the texture coordinate error includes: a first texture coordinate error and/or a second texture coordinate error; the coordinate determination part 2403 is further configured to determine the first texture coordinate error according to the sum of the first distortion measure corresponding to the at least one triangular face;

The second texture coordinate error is determined according to a maximum second distortion measure corresponding to the at least one triangular patch.

In some embodiments, the coordinate determination part 2403 is also configured to update the texture coordinates of the current vertex by minimizing the first texture coordinate error; or, to update the texture coordinates of the current vertex by minimizing the second texture coordinate error; or, to update the texture coordinates of the current vertex by minimizing the first texture coordinate error and the second texture coordinate error respectively.

In some embodiments, the coordinate determination part 2403 is also configured to determine an initial center position point and a search range based on the current vertex; perform at least one search based on the initial center position point and the search range to determine at least one initial target position point; determine a target position point based on a minimum candidate texture coordinate error corresponding to at least one initial target position point; and update the texture coordinates of the current vertex based on the texture coordinates of the target position point.

In some embodiments, the coordinate determination part 2403 is also configured to determine at least one candidate position point corresponding to the current vertex; determine the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point; and determine the search range based on the initial center position point.

In some embodiments, the at least one candidate position point includes: a point at an average position of adjacent points of the current vertex, and/or a midpoint of an adjacent edge of the current vertex.

In some embodiments, the coordinate determination part 2403 is also configured to determine at least one candidate texture coordinate error corresponding to the at least one candidate position point; and determine the initial center position point based on the minimum value of the at least one candidate texture coordinate error and the texture coordinate error corresponding to the current vertex.

In some embodiments, the coordinate determination part 2403 is further configured to determine the search range with the initial center position point as the center and the minimum value of at least one distance from the initial center position point to the third edge of at least one adjacent face as the radius. In some embodiments, the coordinate determination part 2403 is further configured to determine the search direction of each search; based on the initial center position point, according to the search direction of each search, a binary search is performed within the search range to determine the initial target position point corresponding to each search, until at least one search is completed to determine the at least one initial target position point.

In some embodiments, the coordinate determination part 2403 is also configured to use the initial center position point as the starting point of each search, and determine the end point of each search within the search range according to the search direction of each search; when the candidate texture coordinate error corresponding to the starting point is less than the candidate texture coordinate error corresponding to the end point, update the end point according to the midpoint between the starting point and the end point; when the candidate texture coordinate error corresponding to the starting point is greater than the corresponding candidate texture coordinate error of the end point, update the starting point according to the midpoint between the starting point and the end point; perform at least one update until it is determined that the distance between the updated starting point and the updated end point is less than a preset distance threshold, and determine the initial target position point according to the starting point and/or the end point.

In some embodiments, the coordinate determination part 2403 is also configured to determine the starting point as the initial target position point; or, determine the ending point as the initial target position point; or, determine the midpoint between the starting point and the ending point as the initial target position point; or, determine the point between the starting point and the ending point with the smallest error in the candidate texture coordinates as the target position point.

In some embodiments, the coordinate determination part 2403 is further configured to determine at least one texture coordinate error corresponding to the at least one vertex; organize the at least one vertex into a sequence to be optimized based on the at least one texture coordinate error; determine the vertex corresponding to a preset sequence position in the sequence to be optimized as the current vertex; the preset sequence position corresponds to the vertex with the largest texture coordinate error among the at least one vertex.

In some embodiments, the coordinate determination part 2403 is also configured to update the texture coordinate error corresponding to the current vertex in the at least one vertex according to the target position point, and then update the order of the vertices in the sequence to be optimized; update the texture coordinates of the updated vertices at the preset sequence position until the preset iteration target is reached, thereby completing the reconstruction of the texture coordinate information of the deformed mesh.

In some embodiments, the preset iteration target includes:

The update times of the sequence to be optimized reaches a preset update times threshold; and/or the texture coordinate error of the vertex to be optimized at the preset sequence position is less than a preset error threshold.

In some embodiments, the reconstructed deformed mesh includes: the i-th layer reconstructed deformed mesh corresponding to the first basic mesh; i is greater than 1, and the coordinate determination part 2403 is further configured to, after completing the update of the texture coordinates of the i-th layer reconstructed deformed mesh, interpolate the texture coordinates of the vertices in the i-th layer reconstructed deformed mesh to determine the texture coordinates of the vertices in the i+1-th layer reconstructed deformed mesh; i+1 is less than or equal to a preset number of layers; by determining the first distortion measure and/or the second distortion measure of the triangle facets corresponding to the vertices in the i+1-th layer reconstructed deformed mesh, update the texture coordinates of the vertices in the i+1-th layer reconstructed deformed mesh.

It can be understood that in this embodiment, "part" can be a part of the circuit, a part of the processor, a part of the program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional modules.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer-readable storage medium, which is applied to the decoder 240, and the computer-readable storage medium stores a computer program. When the computer program is executed by the second processor, the method described in any one of the above embodiments is implemented.

It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

Based on the composition of the decoder 240 and the computer-readable storage medium, refer to Figure 25, which shows a specific hardware structure diagram of the decoder 240 provided in an embodiment of the present application. As shown in Figure 25, the decoder 240 may include: a second communication interface 2501, a second memory 2502 and a second processor 2503; each component is coupled together through a second bus system 2504. It can be understood that the second bus system 2504 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 2504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as the second bus system 2504 in Figure 25. Among them,

The second communication interface 2501 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

The second memory 2502 is used to store a computer program that can be run on the second processor 2503;

The second processor 2503 is used to execute the decoding method applied to the decoder provided in the embodiment of the present application when running the computer program.

Optionally, as another embodiment, the second processor 2503 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

It can be understood that the hardware functions of the second memory 2502 and the first memory 2302 are similar, and the hardware functions of the second processor 2503 and the first processor 2303 are similar; they will not be described in detail here.

In yet another embodiment of the present application, referring to FIG26 , a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application is shown. As shown in FIG26 , a coding and decoding system 260 may include an encoder 2601 and a decoder 2602 .

In the embodiment of the present application, the encoder 2601 may be the encoder described in any one of the aforementioned embodiments, and the decoder 2602 may be the decoder described in any one of the aforementioned embodiments.

It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Industrial Applicability

The embodiment of the present application provides a coding and decoding method, a code stream, an encoder, a decoder and a storage medium. At the decoding end, the code stream is parsed to determine the first basic grid and the texture coordinate information of the first basic grid; the grid is subdivided and deformed based on the first basic grid to determine the reconstructed deformed grid; the texture coordinate information of the reconstructed deformed grid is determined based on the texture coordinate information of the first basic grid and the reconstructed deformed grid obtained by subdividing and deforming, thereby reducing the distortion of the texture coordinates of the vertices, improving the accuracy of the texture coordinates and the matching degree between the reconstructed deformed grid, thereby improving the quality of the reconstructed deformed grid and the decoding performance. Furthermore, the distortion of the texture coordinates of the vertices in the reconstructed deformed grid is measured by two measures: the average stretching of the surface local distance of the triangular face from the three-dimensional coordinate space to the texture coordinate space and/or the worst-case stretching of the triangular face from the three-dimensional coordinate space to the texture coordinate space, and the texture coordinates of the vertices are adjusted according to the distortion, the texture coordinates of the vertices are updated, the accuracy of the texture coordinates and the matching degree between the reconstructed deformed grid are improved, thereby improving the quality of the reconstructed deformed grid, and then improving the decoding performance. Furthermore, by separately encoding the texture coordinate information through the encoder and sending it to the decoder, the distortion of the texture coordinate information is reduced, the encoding and decoding accuracy is further improved, and thus the encoding and decoding performance is improved.

Claims (36)

A decoding method, applied to a decoder, comprising: Parse the code stream to determine the first basic grid and texture coordinate information of the first basic grid; Performing mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh; The texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh. The method according to claim 1, wherein the parsing the bitstream to determine the first base mesh and the texture coordinate information of the first base mesh comprises: Demultiplexing the code stream to determine a first basic grid code stream; The first basic grid code stream is parsed to determine the first basic grid including the texture coordinate information, thereby determining the first basic grid and the texture coordinate information of the first basic grid. The method according to claim 1, wherein the parsing the bitstream to determine the first base mesh and the texture coordinate information of the first base mesh comprises: Demultiplexing the code stream to determine a second basic grid code stream and a texture coordinate code stream; Parsing the second basic grid code stream to determine a second basic grid; Parsing the texture coordinate code stream to determine texture coordinate information; The second basic mesh is reconstructed according to the texture coordinate information to determine the first basic mesh and the texture coordinate information of the first basic mesh. The method according to claim 3, wherein the reconstructing the second base mesh according to the texture coordinate information and determining the first base mesh and the texture coordinate information of the first base mesh comprises: Reconstructing the second basic mesh according to the texture coordinate information to determine a basic mesh including the texture coordinate information; Mesh cleaning is performed on the basic mesh including the texture coordinate information to determine the first basic mesh and the texture coordinate information of the first basic mesh. The method according to claim 1, wherein the performing mesh subdivision and deformation based on the first basic mesh to determine the reconstructed deformed mesh comprises: Performing mesh subdivision based on the first basic mesh to determine a basic subdivided mesh; The reconstructed deformed grid is determined based on the basic subdivided grid and the displacement information parsed from the bitstream. The method according to claim 5, wherein the determining the texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh comprises: Determining initial texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the basic subdivided mesh; Based on the initial texture coordinate information of the reconstructed deformed mesh, a first distortion measure and/or a second distortion measure is determined; the first distortion measure represents an average stretching of a surface local distance of a triangular facet in the reconstructed deformed mesh from a three-dimensional coordinate space to a texture coordinate space; the second distortion measure represents a worst-case stretching of a triangular facet in the reconstructed deformed mesh from a three-dimensional coordinate space to a texture coordinate space; Texture coordinate information of the reconstructed deformed mesh is determined according to the first distortion measure and/or the second distortion measure. The method according to claim 6, wherein determining the first distortion measure and/or the second distortion measure based on the initial texture coordinate information of the reconstructed deformed mesh comprises: Determining texture coordinates and three-dimensional coordinates of triangles corresponding to vertices in the reconstructed deformed mesh based on initial texture coordinate information of the reconstructed mesh and three-dimensional coordinate information of the reconstructed mesh; Determine a first length parameter and a second length parameter according to the texture coordinates and the three-dimensional coordinates of the triangular face; the first length parameter represents a minimum length corresponding to the mapping of a unit length vector from the texture coordinate space to the three-dimensional coordinate space; the second length parameter represents a maximum length corresponding to the mapping of a unit length vector from the texture coordinate space to the three-dimensional coordinate space; Determining the first distortion measure according to the first length parameter and the second length parameter; and/or, The second distortion measure is determined based on the second length parameter. The method according to claim 7, wherein determining the first length parameter and the second length parameter according to the texture coordinates and the three-dimensional coordinates of the triangular facet comprises: Determining a first partial derivative and a second partial derivative according to an affine mapping between the texture coordinates of the triangular patch and the three-dimensional coordinates; Determine a partial derivative matrix corresponding to the vertices in the triangular patch according to the first partial derivative and the second partial derivative; Determining the first length parameter according to a first singular value corresponding to the partial derivative matrix; The second length parameter is determined according to a second singular value corresponding to the partial derivative matrix; the first singular value is greater than the second singular value. The method according to claim 8, wherein the method further comprises: The first distortion measure is determined according to a first dot product corresponding to the first partial derivative and a second dot product corresponding to the second partial derivative. The method according to any one of claims 6 to 9, wherein determining the texture coordinate information of the reconstructed deformed mesh according to the first distortion measure and/or the second distortion measure comprises: determining, according to the first distortion measure and/or the second distortion measure, a texture coordinate error corresponding to a current vertex in the reconstructed deformed mesh; The texture coordinates of the current vertex are updated by minimizing the texture coordinate error, thereby determining the texture coordinate information of the reconstructed deformed mesh. The method according to claim 10, wherein the triangular facet corresponding to the current vertex includes: at least one triangular facet adjacent to the current vertex; the texture coordinate error includes: a first texture coordinate error and/or a second texture coordinate error; and determining the texture coordinate error corresponding to the current vertex according to the first distortion measure and/or the second distortion measure comprises: determining the first texture coordinate error according to a sum of first distortion measures corresponding to the at least one triangular patch; and/or, The second texture coordinate error is determined according to a maximum second distortion measure corresponding to the at least one triangular patch. The method according to claim 11, wherein updating the texture coordinates of the current vertex by minimizing the texture coordinate error comprises: Updating the texture coordinates of the current vertex by minimizing the first texture coordinate error; or, Updating the texture coordinates of the current vertex by minimizing the second texture coordinate error; or, The texture coordinates of the current vertex are updated by minimizing the first texture coordinate error and the second texture coordinate error respectively. The method according to claim 10, wherein updating the texture coordinates of the current vertex by minimizing the texture coordinate error comprises: Based on the current vertex, determine an initial center position point and a search range; Perform at least one search based on the initial center position point and the search range to determine at least one initial target position point; Determining a target position point according to a minimum candidate texture coordinate error corresponding to at least one initial target position point; The texture coordinates of the current vertex are updated according to the texture coordinates of the target position point. The method according to claim 10, wherein determining the initial center position point and the search range based on the current vertex comprises: Determine at least one candidate position point corresponding to the current vertex; Determine the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point; Based on the initial central position point, the search range is determined. The method according to claim 14, wherein the at least one candidate location point comprises: A point at the average position of the adjacent points of the current vertex, and/or a midpoint of the adjacent edge of the current vertex. The method according to claim 14, wherein determining the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point comprises: Determining at least one candidate texture coordinate error corresponding to the at least one candidate position point; The initial center position point is determined according to a minimum value between the at least one candidate texture coordinate error and the texture coordinate error corresponding to the current vertex. The method according to claim 14, wherein determining the search range based on the initial center position point comprises: The search range is determined by taking the initial central position point as the center and taking the minimum value of at least one distance from the initial central position point to the third edge of at least one adjacent face as the radius. The method according to claim 13, wherein the determining at least one initial target location point based on the initial center location point and the search range by performing at least one search comprises: Determine the search direction for each search; Based on the initial center position point, a binary search is performed within the search range according to the search direction of each search to determine the initial target position point corresponding to each search, until at least one search is completed to determine the at least one initial target position point. The method according to claim 18, wherein the step of performing a binary search within the search range based on the initial center position point and in accordance with the search direction of each search to determine the initial target position point corresponding to each search comprises: Taking the initial center position point as the starting point of each search, and determining the end point of each search within the search range according to the search direction of each search; When the error of the candidate texture coordinates corresponding to the starting point is smaller than the error of the candidate texture coordinates corresponding to the ending point, updating the ending point according to the midpoint between the starting point and the ending point; When the error of the candidate texture coordinates corresponding to the starting point is greater than the error of the candidate texture coordinates corresponding to the ending point, updating the starting point according to the midpoint between the starting point and the ending point; Perform at least one update until it is determined that the distance between the updated starting point and the updated ending point is less than a preset distance threshold, and determine the initial target location point according to the starting point and/or the ending point. The method according to claim 19, wherein determining the initial target location point according to the starting point and/or the ending point comprises: Determine the starting point as the initial target location point; or, Determine the end point as the initial target location point; Alternatively, the midpoint between the starting point and the ending point is determined as the initial target position point; Alternatively, a point between the starting point and the ending point with a smaller error in candidate texture coordinates is determined as the target position point. The method according to any one of claims 11 to 20, wherein the method further comprises: determining at least one texture coordinate error corresponding to the at least one vertex; organizing the at least one vertex into a sequence to be optimized according to the at least one texture coordinate error; The vertex corresponding to a preset sequence position in the sequence to be optimized is determined as the current vertex; the preset sequence position corresponds to the vertex with the largest texture coordinate error among the at least one vertex. The method according to claim 21, wherein determining texture coordinate information of the reconstructed deformed mesh comprises: According to the target position point, updating the texture coordinate error corresponding to the current vertex in the at least one vertex, and then updating the order of the vertices in the sequence to be optimized; The texture coordinates of the updated vertices at the preset sequence positions are updated until a preset iteration target is reached, thereby completing the reconstruction of the texture coordinate information of the deformed mesh. The method according to claim 21, wherein the preset iteration target comprises: The update times of the sequence to be optimized reaches a preset update times threshold; And/or, the texture coordinate error of the vertices at the preset sequence positions is less than a preset error threshold. The method according to claim 6 or 22, wherein the reconstructed deformed mesh comprises: an i-th layer reconstructed deformed mesh corresponding to the first basic mesh; i is greater than 1, and the method further comprises: When the texture coordinates of the i-th layer of the reconstructed deformed mesh are updated, interpolation is performed based on the texture coordinates of the vertices in the i-th layer of the reconstructed deformed mesh to determine the texture coordinates of the vertices in the i+1-th layer of the reconstructed deformed mesh; i+1 is less than or equal to the preset number of layers; The texture coordinates of the vertices in the (i+1)th layer reconstructed deformed mesh are updated by determining the first distortion measure and/or the second distortion measure of the triangles corresponding to the vertices in the (i+1)th layer reconstructed deformed mesh. The method according to claim 24, wherein the method further comprises: The current image is restored by updating texture coordinates of at least one layer of reconstructed deformed mesh corresponding to the first basic mesh. A coding method, applied to an encoder, comprising: Determine the base grid corresponding to the current image; Based on the base mesh, determining texture coordinate information; The basic grid and the texture coordinate information are encoded, and the obtained encoding bits are written into a bit stream. The method according to claim 26, wherein the texture coordinate information comprises: texture coordinate information corresponding to the base mesh; and determining the texture coordinate information based on the base mesh comprises: Mesh parameterization is performed based on the basic mesh to determine the texture coordinate information. The method according to claim 24, wherein the texture coordinate information comprises: texture coordinate information corresponding to the reconstructed base mesh; and determining the texture coordinate information based on the base mesh comprises: Reconstructing based on the coded bits of the basic grid to determine a reconstructed basic grid; Mesh parameterization is performed on the reconstructed basic mesh to determine the texture coordinate information. The method according to claim 27, wherein encoding the base mesh and the texture coordinate information and writing the obtained encoded bits into a bitstream comprises: The base grid including the texture coordinate information is encoded, and the obtained encoding bits are written into a bitstream. The method according to claim 27 or 28, wherein encoding the base mesh and the texture coordinate information and writing the obtained encoded bits into a bitstream comprises: Performing compression coding on the basic grid to determine coding bits corresponding to the basic grid; Encoding the texture coordinate information to determine encoding bits corresponding to the texture coordinate information; The coding bits corresponding to the basic grid and the coding bits corresponding to the texture coordinate information are written into the bitstream respectively. A decoder, comprising: A parsing part, configured to parse the code stream to determine the first basic grid and texture coordinate information of the first basic grid; A subdivision and deformation part is configured to perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh; The coordinate determination part is configured to determine the texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh. An encoder, comprising: The grid information determination part is configured to determine a basic grid corresponding to the current image; and determine texture coordinate information based on the basic grid; The encoding part is configured to encode the basic grid and the texture coordinate information, and write the obtained encoding bits into a bit stream. A decoder, comprising: a first memory and a first processor; The first memory stores a computer program executable on the first processor, and the first processor implements the method according to any one of claims 1 to 25 when executing the program. An encoder, comprising: a second memory and a second processor; The second memory stores a computer program that can be run on the second processor, and the second processor implements the method according to any one of claims 26 to 30 when executing the program. A code stream, wherein the code stream is generated by bit coding according to information to be coded; wherein the information to be coded at least includes: basic grid and texture coordinate information. A storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 25 is implemented, or the method according to any one of claims 26 to 30 is implemented.