WO2025010558A1 - Video encoding method and apparatus, video decoding method and apparatus, device, system, and storage medium - Google Patents

Abstract

The present application provides a video encoding method and apparatus, a video decoding method and apparatus, a device, a system, and a storage medium. The video encoding method comprises: when encoding N views, determining a first pruning mask map of an i-th view; on the basis of a preset block division mode, dividing the i-th view into M image blocks, for a j-th image block among the M image blocks, determining unpruned pixel points in the j-th image block on the basis of the first pruning mask map, and performing chromaticity fitting on the unpruned pixel points in the j-th image block to obtain color cast fitting functions of the j-th image block; pruning the unpruned pixel points in the j-th image block on the basis of the color cast fitting functions to obtain patch information of the i-th view; and encoding the patch information and the color cast fitting function to obtain a bitstream. That is, according to the present application, the first pruning mask map is re-pruned to reduce the number of unpruned pixel points, thereby reducing the encoding cost of an encoding end, and improving the multi-view video encoding efficiency.

Description

Video encoding and decoding method, device, equipment, system, and storage medium Technical Field

The present application relates to the field of video coding and decoding technology, and in particular to a video coding and decoding method, device, equipment, system, and storage medium.

Background Art

In three-dimensional application scenarios, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), multi-viewpoint videos are used to provide users with an immersive experience.

In the current multi-view video encoding process, in order to reduce the amount of encoded data, the redundant information between the multi-view videos is pruned by pixel pruning to reduce the amount of encoded data. However, the current pixel pruning method has the problem of incomplete pruning, which makes the encoding cost still high.

Summary of the invention

The embodiments of the present application provide a video encoding and decoding method, apparatus, device, system, and storage medium, which can achieve complete cropping of pixels, thereby reducing the encoding cost and improving the encoding and decoding efficiency of multi-viewpoint videos.

In a first aspect, the present application provides a video decoding method, applied to a decoder, comprising:

For an i-th view among N views, decode a bitstream, determine patch information of the i-th view, and generate a patch image of the i-th view based on the patch information, wherein the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N;

According to a preset block division method, the patch image is divided into M image blocks, where M is a positive integer;

For a j-th image block among the M image blocks, decode the bitstream to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer;

Using the P color deviation fitting functions, pixel fitting is performed on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block.

In a second aspect, an embodiment of the present application provides a video encoding method, applied to an encoder, comprising:

For an i-th view among N views, determine a first pruning mask map of the i-th view, where the first pruning mask map is a pruning mask map obtained by performing pixel pruning on the i-th view, where the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N;

Based on a preset block division method, the i-th view is divided into M image blocks, where M is a positive integer;

For a j-th image block among the M image blocks, determine unpruned pixels in the j-th image block based on the first pruning mask image, and perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer;

Pruning unpruned pixels in the j-th image block based on the P color cast fitting functions to obtain patch information of the i-th view;

The patch information and the color cast fitting function are encoded to obtain a bit stream.

In a third aspect, the present application provides a video decoding device, which is used to execute the method in the first aspect or its respective implementations. Specifically, the device includes a functional unit for executing the method in the first aspect or its respective implementations.

In a fourth aspect, the present application provides a video encoding device, which is used to execute the method in the second aspect or its respective implementations. Specifically, the device includes a functional unit for executing the method in the second aspect or its respective implementations.

In a fifth aspect, the present application provides a video decoder, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the first aspect or its implementations.

In a sixth aspect, the present application provides a video encoder, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the second aspect or its implementations.

In a seventh aspect, the present application provides a video coding and decoding system, including a video encoder and a video decoder. The video decoder is used to execute the method in the first aspect or its respective implementations, and the video encoder is used to execute the method in the second aspect or its respective implementations.

In an eighth aspect, the present application provides a chip for implementing the method in any one of the first to second aspects or their respective implementations. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in any one of the first to second aspects or their respective implementations.

In a ninth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the method of any one of the first to second aspects above or any of their implementations.

In a tenth aspect, the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method in any one of the first to second aspects above or in each of their implementations.

In an eleventh aspect, the present application provides a computer program, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects or in each of their implementations.

Based on the above technical solution, when encoding a multi-view video, firstly, based on the pixel pruning method, a first pruning mask map of the i-th view among N views is determined. Then, based on a preset block division method, the i-th view is divided into M image blocks. For the j-th image block among the M image blocks, the unpruned pixels in the j-th image block are determined based on the first pruning mask map, and the unpruned pixels in the j-th image block are chromatically fitted to obtain a color deviation fitting function of the j-th image block. Then, the unpruned pixels in the j-th image block are pruned based on the color deviation fitting function to obtain patch information of the i-th view. Finally, the above patch information and the color deviation fitting function are encoded to obtain a bitstream. That is, in the embodiment of the present application, the first pruning mask map obtained by pixel pruning is pruned again based on the color deviation fitting function to further reduce the number of unpruned pixels, reduce the amount of data that needs to be encoded at the encoding end, and thereby reduce the encoding cost of the encoding end, and improve the encoding and decoding efficiency of the multi-view video.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application;

FIG2A is a schematic block diagram of a video encoder according to an embodiment of the present application;

FIG2B is a schematic block diagram of a video decoder according to an embodiment of the present application;

FIG3A schematically shows a schematic diagram of three degrees of freedom;

FIG3B schematically shows a schematic diagram of three degrees of freedom +;

FIG3C schematically shows a schematic diagram of six degrees of freedom;

4A and 4B are schematic diagrams showing the principle of MIV technology;

FIG5A is a schematic diagram of a multi-view video encoding process;

FIG5B is a schematic diagram of a multi-view video decoding process;

FIG6 is a schematic diagram of the encoding process of TMIV14;

FIG7A is a schematic diagram of a view list;

FIG7B is a schematic diagram of an aggregate of trimmed views;

FIG7C is a schematic diagram of patch packaging;

FIG8 is a schematic diagram of a video decoding method flow chart provided by an embodiment of the present application;

FIG9 is a schematic diagram of a patch image;

FIG10 is a schematic diagram of dividing a patch image into blocks;

FIG11 is a schematic diagram of a directed acyclic graph;

FIG12 is a schematic diagram of decoding;

FIG13 is a schematic diagram of a video encoding method flow chart provided by an embodiment of the present application;

FIG14A is a schematic diagram of a view from different viewpoints;

FIG14B is a schematic diagram of pixel projection;

FIG14C is a schematic diagram of pixel pruning;

15 and 16 are schematic diagrams of an encoding process involved in an embodiment of the present application;

FIG17 is a schematic block diagram of a video decoding device provided by an embodiment of the present application;

FIG18 is a schematic block diagram of a video encoding device provided by an embodiment of the present application;

FIG19 is a schematic block diagram of an electronic device provided in an embodiment of the present application;

FIG. 20 is a schematic block diagram of a video encoding and decoding system provided in an embodiment of the present application.

DETAILED DESCRIPTION

The present application can be applied to the field of image coding and decoding, the field of video coding and decoding, the field of hardware video coding and decoding, the field of dedicated circuit video coding and decoding, the field of real-time video coding and decoding, etc. For example, the solution of the present application can be combined with an audio and video coding standard (AVS), such as the H.264/audio and video coding (AVC) standard, the H.265/high efficiency video coding (HEVC) standard, and the H.266/versatile video coding (VVC) standard. Alternatively, the solution of the present application may be combined with other proprietary or industry standards and operate, the standards include ITU-TH.261, ISO/IEC MPEG-1 Visual, ITU-TH.262 or ISO/IEC MPEG-2 Visual, ITU-TH.263, ISO/IEC MPEG-4 Visual, ITU-TH.264 (also known as ISO/IEC MPEG-4 AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any particular codec standard or technology.

The high-degree-of-freedom immersive coding system can be roughly divided into the following links according to the task line: data collection, data organization and expression, data encoding and compression, data decoding and reconstruction, data synthesis and rendering, and finally presenting the target data to the user.

The encoding involved in the embodiment of the present application is mainly video encoding and decoding. For ease of understanding, the video encoding and decoding system involved in the embodiment of the present application is first introduced in conjunction with Figure 1.

FIG1 is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that FIG1 is only an example, and the video encoding and decoding system of the embodiment of the present application includes but is not limited to that shown in FIG1. As shown in FIG1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compression) the video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, etc.

In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via the channel 130. The channel 130 may include one or more media and/or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.

In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to the communication standard and transmit the modulated video data to the decoding device 120. The communication medium includes a wireless communication medium, such as a radio frequency spectrum, and optionally, the communication medium may also include a wired communication medium, such as one or more physical transmission lines.

In another example, the channel 130 includes a storage medium, which can store the video data encoded by the encoding device 110. The storage medium includes a variety of locally accessible data storage media, such as optical disks, DVDs, flash memories, etc. In this example, the decoding device 120 can obtain the encoded video data from the storage medium.

In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In the example, the decoding device 120 can download the stored encoded video data from the storage server. Optionally, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator/demodulator (modem) and/or a transmitter.

In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .

The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

The video encoder 112 encodes the video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the encoding information of the picture or the sequence of pictures in the form of a bitstream. The encoding information may include the encoded picture data and associated data. The associated data may include a sequence parameter set (SPS for short), a picture parameter set (PPS for short) and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .

In some embodiments, the decoding device 120 may include a display device 123 in addition to the input interface 121 and the video decoder 122 .

The input interface 121 includes a receiver and/or a modem. The input interface 121 can receive the encoded video data through the channel 130 .

The video decoder 122 is used to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .

The display device 123 displays the decoded video data. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

In addition, FIG1 is only an example, and the technical solution of the embodiment of the present application is not limited to FIG1 . For example, the technology of the present application can also be applied to unilateral video encoding or unilateral video decoding.

The following is an introduction to the video encoding framework involved in the embodiments of the present application.

FIG2A is a schematic block diagram of a video encoder according to an embodiment of the present application. It should be understood that the video encoder 200 can be used for lossy compression of an image, or lossless compression of an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

The video encoder 200 can be applied to image data in luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2 or 4:4:4, Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

For example, the video encoder 200 reads video data, and for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, CTB may be referred to as a "tree block", "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chrominance or chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding, and a CU may be a rectangular block or a square block. CU can be further divided into prediction unit (PU) and transform unit (TU), which makes encoding, prediction and transform separated and more flexible in processing. In one example, CTU is divided into CU in quadtree mode, and CU is divided into TU and PU in quadtree mode.

The video encoder and video decoder may support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder may support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N or similar sizes for inter-frame prediction. The video encoder and video decoder may also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

In some embodiments, as shown in FIG2A , the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform/quantization unit 230, an inverse transform/quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer, or different functional components.

Optionally, in the present application, the current block may be referred to as a current coding unit (CU) or a current prediction unit (PU), etc. A prediction block may also be referred to as a prediction image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame estimation unit 212. Since there is a strong correlation between adjacent pixels in a frame of a video, an intra-frame prediction method is used in the video coding and decoding technology to eliminate spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent frames in a video, an inter-frame prediction method is used in the video coding and decoding technology to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

The inter-frame prediction unit 211 can be used for inter-frame prediction. Inter-frame prediction may include motion estimation and motion compensation. It may refer to image information of different frames. Inter-frame prediction uses motion information to find reference blocks from reference frames, and generates prediction blocks based on the reference blocks to eliminate temporal redundancy. The frames used for inter-frame prediction may be P frames and/or B frames. P frames refer to forward prediction frames, and B frames refer to bidirectional prediction frames. Inter-frame prediction uses motion information to find reference blocks from reference frames, and generates prediction blocks based on the reference blocks. Motion information includes a reference frame list where the reference frame is located, a reference frame index, and a motion vector. The motion vector may be an integer pixel or a sub-pixel. If the motion vector is a sub-pixel, an interpolation filter is required in the reference frame to make the required sub-pixel block. Here, the integer pixel or sub-pixel block in the reference frame found according to the motion vector is called a reference block. Some technologies will directly use the reference block as the prediction block, while some technologies will generate a prediction block based on the reference block. Generating a prediction block based on the reference block can also be understood as using the reference block as the prediction block and then generating a new prediction block based on the prediction block.

The intra-frame estimation unit 212 only refers to the information of the same frame image to predict the pixel information in the current code image block to eliminate spatial redundancy. The frame used for intra-frame prediction can be an I frame.

There are multiple prediction modes for intra-frame prediction. Taking the H series of international digital video coding standards as an example, the H.264/AVC standard has 8 angle prediction modes and 1 non-angle prediction mode, and H.265/HEVC is extended to 33 angle prediction modes and 2 non-angle prediction modes. The intra-frame prediction modes used by HEVC are Planar, DC, and 33 angle modes, for a total of 35 prediction modes. The intra-frame modes used by VVC are Planar, DC, and 65 angle modes, for a total of 67 prediction modes.

It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital videos.

The residual unit 220 may generate a residual block of the CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, the residual unit 220 may generate a residual block of the CU so that each sample in the residual block has a value equal to the difference between the following two: a sample in the pixel blocks of the CU and a corresponding sample in the prediction blocks of the PUs of the CU.

The transform/quantization unit 230 may quantize the transform coefficients. The transform/quantization unit 230 may quantize the transform coefficients associated with the TUs of the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

The inverse transform/quantization unit 240 may apply inverse quantization and inverse transform to the quantized transform coefficients, respectively, to reconstruct a residual block from the quantized transform coefficients.

The reconstruction unit 250 may add the samples of the reconstructed residual block to the corresponding samples of one or more prediction blocks generated by the prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of the CU in this manner, the video encoder 200 may reconstruct the pixel blocks of the CU.

The loop filter unit 260 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent coded pixels. For example, a deblocking filter operation may be performed to reduce the blocking effect of the pixel blocks associated with the CU.

In some embodiments, the loop filter unit 260 includes a deblocking filter unit and a sample adaptive offset/adaptive loop filter (SAO/ALF) unit, wherein the deblocking filter unit is used to remove the block effect, and the SAO/ALF unit is used to remove the ringing effect.

The decoded image buffer 270 may store the reconstructed pixel blocks. The inter prediction unit 211 may use the reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra estimation unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.

The entropy encoding unit 280 may receive the quantized transform coefficients from the transform/quantization unit 230. The entropy encoding unit 280 may perform one or more entropy encoding operations on the quantized transform coefficients to generate entropy-encoded data.

FIG. 2B is a schematic block diagram of a video decoder according to an embodiment of the present application.

2B , the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization/transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded image buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.

The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the syntax elements in the bitstream that have been entropy encoded. The prediction unit 320, the inverse quantization/transformation unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data according to the syntax elements extracted from the bitstream, that is, generate decoded video data.

In some embodiments, the prediction unit 320 includes an inter-frame prediction unit 321 and an intra-frame estimation unit 322 .

The intra estimation unit 322 may perform intra prediction to generate a prediction block for the PU. The intra estimation unit 322 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra estimation unit 322 may also determine the intra prediction mode for the PU according to one or more syntax elements parsed from the code stream.

The inter prediction unit 321 may construct a first reference image list (list 0) and a second reference image list (list 1) according to the syntax elements parsed from the code stream. In addition, if the PU is encoded using inter prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter prediction unit 321 may determine one or more reference blocks of the PU according to the motion information of the PU. The inter prediction unit 321 may generate a prediction block of the PU according to one or more reference blocks of the PU.

The inverse quantization/transform unit 330 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization/transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization.

After inverse quantizing the transform coefficients, the inverse quantization/transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 may add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking effects of pixel blocks associated with a CU.

The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

The basic process of video encoding and decoding is as follows: at the encoding end, a frame of image is divided into blocks, and for the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block of the current block. The residual unit 220 can calculate the residual block based on the original block of the prediction block and the current block, that is, the difference between the original block of the prediction block and the current block, and the residual block can also be called residual information. The residual block can remove information that is not sensitive to the human eye through the transformation and quantization process of the transformation/quantization unit 230 to eliminate visual redundancy. Optionally, the residual block before transformation and quantization by the transformation/quantization unit 230 can be called a time domain residual block, and the time domain residual block after transformation and quantization by the transformation/quantization unit 230 can be called a frequency residual block or a frequency domain residual block. The entropy coding unit 280 receives the quantized change coefficient output by the change quantization unit 230, and can entropy encode the quantized change coefficient and output a bit stream. For example, the entropy coding unit 280 can eliminate character redundancy according to the target context model and the probability information of the binary bit stream.

At the decoding end, the entropy decoding unit 310 can parse the bitstream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 generates a prediction block of the current block using intra-frame prediction or inter-frame prediction based on the prediction information. The inverse quantization/transformation unit 330 uses the quantization coefficient matrix obtained from the bitstream to The coefficient matrix is inversely quantized and inversely transformed to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks form a reconstructed image, and the loop filter unit 350 performs loop filtering on the reconstructed image based on the image or on the block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be a reference frame for inter-frame prediction for subsequent frames.

It should be noted that the block division information determined by the encoder, as well as the mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder by analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder is the same as the decoded image obtained by the decoder.

The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The present application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

The embodiments of the present application involve encoding and decoding of perspective videos. The following introduces relevant concepts involved in the embodiments of the present application.

Multi-view video, also known as free viewpoint video, is an immersive media video captured by multiple cameras, containing different perspectives and supporting user 3DoF+ or 6DoF interaction.

Degree of Freedom (DoF) in a mechanical system refers to the number of independent coordinates, including the degrees of freedom of translation, rotation and vibration. In the embodiment of the present application, it refers to the degrees of freedom that support movement and generate content interaction when the user is watching immersive media.

Three degrees of freedom (3DoF) refers to the three degrees of freedom of the user's head rotating around the XYZ axis. Figure 3A schematically shows a schematic diagram of three degrees of freedom. As shown in Figure 3A, a certain point in a certain place can rotate on three axes, you can turn your head, lower your head up and down, or shake your head. Through the three-degree-of-freedom experience, users can immerse themselves in a scene 360 degrees. If it is static, it can be understood as a panoramic picture. If the panoramic picture is dynamic, it is a panoramic video, that is, a VR video. However, VR videos have certain limitations. Users cannot move and cannot choose any place to watch.

3DoF+: On the basis of three degrees of freedom, the user also has the freedom to make limited movements along the XYZ axis, which can also be called limited six degrees of freedom, and the corresponding media stream can be called limited six degrees of freedom media stream. Figure 3B schematically shows a schematic diagram of three degrees of freedom+.

6DoF: That is, on the basis of three degrees of freedom, the user also has the freedom to move freely along the XYZ axis, and the corresponding media code stream can be called a six-degree-of-freedom media code stream. Figure 3C schematically shows a schematic diagram of six degrees of freedom. Among them, 6DoF media refers to 6-degree-of-freedom video, which means that the video can provide users with a high-degree-of-freedom viewing experience of freely moving the viewpoint in the XYZ axis direction of three-dimensional space and freely rotating the viewpoint around the XYX axis. 6DoF media is a combination of videos from different perspectives of space acquired by a camera array. In order to facilitate the expression, storage, compression and processing of 6DoF media, 6DoF media data is expressed as a combination of the following information: texture maps acquired by multiple cameras, depth maps corresponding to the texture maps of multiple cameras, and corresponding 6DoF media content description metadata, which contains the parameters of multiple cameras, as well as description information such as the splicing layout and edge protection of 6DoF media. At the encoding end, the texture map information of multiple cameras and the corresponding depth map information are spliced, and the description data of the splicing method is written into the metadata according to the defined syntax and semantics. The stitched multi-camera depth map and texture map information is encoded using a planar video compression method and transmitted to the terminal for decoding. The 6DoF virtual viewpoint requested by the user is then synthesized, thereby providing the user with a 6DoF media viewing experience.

Depth map: As a way to express three-dimensional scene information, the grayscale value of each pixel in the depth map can be used to represent the distance between a certain point in the scene and the camera.

In some embodiments, MPEG (Moving Picture Experts Group) immersive video (MPEG Immersive Video, MIV for short) technology is used for encoding and decoding of multi-viewpoint videos.

The following is an introduction to MIV technology.

MIV technology: In order to reduce the transmission pixel rate while retaining scene information as much as possible, so as to ensure that there is enough information for rendering the target view, the solution adopted by MPEG-I is shown in Figure 4A. A limited number of viewpoints are selected as basic viewpoints and the visible range of the scene is expressed as much as possible. The basic viewpoint is transmitted as a complete image, and the redundant pixels between the remaining non-basic viewpoints and the basic viewpoint are removed, that is, only the effective information that is not expressed repeatedly is retained, and then the sub-block image extracted from the effective information is reorganized with the basic viewpoint image to form a larger rectangular image, which is called a spliced image. Figures 4A and 4B show the schematic process of generating the spliced image. The spliced image is sent to the codec for compression and reconstruction, and the auxiliary data related to the splicing information of the sub-block image is also sent to the encoder to form a bit stream.

In some embodiments, multi-viewpoint video is encoded and decoded using the frame packing technology in Visual Volumetric Video-based Coding (V3C).

The following is an introduction to frame packing technology.

Taking multi-view video as an example, illustratively, as shown in FIG5A , the encoding end includes the following steps:

Step 1, when encoding the acquired multi-view video, after some pre-processing, a patch of the multi-view video is generated, and then the patch of the multi-view video is organized to generate a multi-view video mosaic.

For example, as shown in Figure 5A, a multi-view video is input into TIMV for packaging, and a multi-view video splicing image is output. TIMV is a reference software for MIV. The packaging in the embodiment of the present application can be understood as splicing.

The multi-view video mosaic map includes a multi-view video texture mosaic map and a multi-view video geometry mosaic map, that is, it only includes multi-view video sub-blocks.

Step 2: Input the multi-view video mosaic image into the frame packer, and output the multi-view video mixed mosaic image.

Among them, the multi-view video mixed mosaic image includes a multi-view video texture mixed mosaic image, a multi-view video geometry mixed mosaic image, and a multi-view video texture and geometry mixed mosaic image.

Specifically, as shown in FIG5A , the multi-view video mosaic is frame packed to generate a multi-view video mixed mosaic, and each multi-view video mosaic occupies a region of the multi-view video mixed mosaic. Accordingly, a flag pin_region_type_id_minus2 is transmitted for each region in the bitstream. This flag records the information of whether the current region belongs to a multi-view video texture mosaic or a multi-view video geometric mosaic, and the information needs to be used at the decoding end.

Step 3: Use a video encoder to encode the multi-view video mixed splicing image to obtain a bit stream.

Exemplarily, as shown in FIG5B , the decoding end includes the following steps:

Step 1: When decoding a multi-view video, the acquired code stream is input into a video decoder for decoding to obtain a reconstructed multi-view video mixed splicing image.

Step 2: input the reconstructed multi-view video mixed mosaic image into the frame depacketizer, and output the reconstructed multi-view video mosaic image.

Specifically, first, obtain the flag pin_region_type_id_minus2 from the bitstream, if it is determined that the pin_region_type_id_minus2 is V3C_AVD, it means that the current region is a multi-view video texture mosaic, then the current region is split and output as a reconstructed multi-view video texture mosaic.

If it is determined that pin_region_type_id_minus2 is V3C_GVD, it means that the current region is a multi-view video geometric mosaic, and the current region is split and output as a reconstructed multi-view video geometric mosaic.

Step 3: decode the reconstructed multi-view video mosaic to obtain the reconstructed multi-view video.

Specifically, the multi-view video texture mosaic map and the multi-view video geometric mosaic map are decoded to obtain the reconstructed multi-view video.

The encoding process of TMIV14 is introduced below.

The TMIV encoder has a "group-based" encoder that calls the "single group" encoder shown in Figure 6 for each group.

The multi-view video encoder shown in FIG6 is introduced below:

As shown in Figure 6, the encoder operates on a selected source view of a given group and has the following stages:

1. Get the source code view.

The input to the TMIV encoder consists of a list of source views (Figure 7A). A source view represents a projection of a 3D real or virtual scene. Source views can be equirectangular, perspective, or orthographic projections. Each source view should have at least view parameters (camera intrinsics, camera extrinsics, geometry quantization, etc.). A source view may have a geometry section with range/invalid sampling values. Additionally, a source view may have a texture attributes section. Other optional attributes for each source view are entity mapping and transparency attributes sections. The set of components must be the same for all source views.

2. Automatically select parameters.

to set the atlas parameters, i.e. the number of atlases and the bounding box size of each atlas. Some parameters of the TMIV encoder are automatically calculated based on the camera configuration or the first frame of the most source views.

3. Physical layer separation.

Separation of views into entity layers. When an entity map is provided for the source view, TMIV has the ability to operate in entity coding mode. In this mode, the patches extracted and packed in the atlas have active pixels belonging to a single entity for each patch, so each patch can be labeled with its associated entity ID. This enables selective encoding and/or decoding of entities individually when needed, saving utilized bandwidth and improving quality. If entity coding mode is selected, the source view including the base view (texture attributes and geometry information) is sliced into multiple layers such that each layer includes content belonging to one entity. The following encoding stages are then called independently for each entity in order to prune, aggregate and cluster together the layers from all views belonging to the same entity. The patches of all entities are packaged and combined in a set of atlases.

4. Pixel trimming.

That is, redundant information is pruned. Multi-view presentation of a scene is inherently redundant. The pruner selects which areas of the view can be safely pruned. The pruner operates on a frame-by-frame basis, receiving multiple views with texture attributes and geometry information as well as camera parameters, and outputs a mask image of each source view at the same size. For non-base views, masked pixels are "pruned" and unmasked pixels are "retained". For base views, all pixels are "retained".

5. Pruning mask aggregation.

The pruning mask for each entity is aggregated frame by frame by activating the active samples of the pruning mask in the aggregate pruning mask. The mask is reset at the beginning of each intra-frame period. The process is completed by outputting the final aggregated result at the end of the intra-frame period. Figure 7 illustrates the pruning view at frame i, and the aggregation of the active samples (drawn in white) between frame i and frame i+k within the intra-frame period; it can be seen that the contours of the changing parts of the geometric components are getting thicker to take into account the motion in the scene.

6. Cluster active pixels.

A cluster is a connected set of pixels that are active in the aggregate mask of an entity. The connectivity criterion for a pixel is that it has at least one other pixel among its eight neighbors.

An example of clustering, where each cluster of pruned views is represented by a specific false color. The clusters are then sorted in decreasing order of size.

7. Cluster splitting.

In order to reduce the spatial redundancy of the data in the atlas, irregularly shaped clusters are split. Each cluster is split into two smaller clusters if the total area of the bounding boxes of the two newly obtained clusters is smaller than the area of the bounding box of the initial cluster by a threshold. To decide how to split the cluster, the total area of the bounding boxes of the two subclusters is minimized. The split is performed along lines parallel to the short sides of the cluster bounding boxes. This approach allows the partitioning of L-shaped clusters.

8. Patch packaging.

As shown in Figure 7C, the packing process packs each cluster into the atlas in sequence.

9. Modification of patch average value.

After packing the patches into the atlas, all attribute patch values are modified to reduce the number and size of edges between adjacent patches and between occupied and unoccupied regions in the attribute atlas. The mean of each part of the patch is set to a neutral color, ^2m –1 for m-bps video. To restore the original attribute values at the decoder side, patch attribute offsets are added and sent in the atlas data.

10. Color correction.

TMIV has the ability to align the different color characteristics of the source views. If the optional color correction is enabled, the color characteristics of each source view will be aligned with the color characteristics of a reference view that corresponds to the view captured by the camera closest to the center of the camera rig.

11. Video data generation.

The last operation within the single-group encoder is to write the patch to the buffer (geometry and attribute information) of the atlas assigned to it. Note that for entity encoding mode, the content of a given patch is extracted from the associated entity view generated by the entity delimiter based on the entity ID of the patch. This ensures that a patch with the correct entity content (texture attributes and geometry) is written to the formed atlas.

12. Geometric quantization.

Atlas values are either "invalid/unoccupied" or a geometry value expressed in meters, with a maximum geometry value set to 1 km. The MIV specification [2] specifies how occupancy information is encoded in the geometry atlas if occupancy is not explicitly present. Decoding is based on normalized disparity range, geometry thresholds, etc. These values are signaled per view or even per patch.

13. Geometric scaling.

To maximize the efficiency of geometry encoding, the entire range of min-max normalized depth values is divided into a predefined number of equal intervals (set in the encoder configuration), and each interval is adaptively scaled based on the importance of the geometry samples in the corresponding interval. For example, the original geometry values in each interval are mapped to the corresponding scaled geometry range.

14. Occupancy scaling.

If the TMIV encoder is configured to output occupancy video data (instead of embedding occupancy information in the geometry video data), the full-resolution occupancy map is scaled down by a configurable scaling factor. The default factor is the native resolution of the occupancy map. For entity-based encoding, higher resolutions are recommended. The decoder reconstructs the full-resolution occupancy map by performing upscaling using nearest-neighbor interpolation. Note that for complete atlases (e.g., atlases including only the base view), an occupancy map may not be output because all pixels are occupied.

After the above encoding process, atlas data, geometric video data, attribute video data, occupancy video data, etc. are generated.

Exemplary, V3C unit header semantics, as shown in Table 1:

Table 1

As can be seen from the above, in the current multi-view video encoding process, in order to reduce the amount of encoded data, the redundant information between the multi-view videos is pruned by pixel pruning to reduce the amount of encoded data. However, the current pixel pruning method has the problem of incomplete pruning, which makes the encoding cost still high.

In order to solve the above technical problems, in an embodiment of the present application, when encoding a multi-view video, firstly, based on a pixel pruning method, a first pruning mask map of the i-th view among N views is determined. Then, based on a preset block division method, the i-th view is divided into M image blocks, and for the j-th image block among the M image blocks, the unpruned pixels in the j-th image block are determined based on the first pruning mask map, and the unpruned pixels in the j-th image block are chromatically fitted to obtain a color deviation fitting function of the j-th image block. Then, the unpruned pixels in the j-th image block are pruned based on the color deviation fitting function to obtain patch information of the i-th view. Finally, the patch information and the color deviation fitting function are encoded to obtain a bitstream. That is, in an embodiment of the present application, based on the color deviation fitting function, the first pruning mask map obtained by pixel pruning is pruned again to further reduce the number of unpruned pixels, reduce the amount of data that needs to be encoded at the encoding end, thereby reducing the encoding cost of the encoding end and improving the encoding and decoding efficiency of the multi-view video.

In conjunction with FIG. 8 , the video decoding method provided in the embodiment of the present application is introduced by taking the decoding end as an example.

FIG8 is a schematic flow chart of a video decoding method provided by an embodiment of the present application, and the embodiment of the present application is applied to the video decoders shown in FIG1 and FIG3. As shown in FIG8, the method of the embodiment of the present application includes:

S101. For an i-th view among N views, decode a bitstream, determine patch information of the i-th view, and generate a patch image of the i-th view based on the patch information.

In the embodiment of the present application, the N views are views from N different viewpoints, that is, the viewpoints corresponding to the N views are all different.

Wherein N is a positive integer greater than 1. That is, the N views in the embodiment of the present application include 2 views, 3 views, and any number of views greater than 2.

In some embodiments, the N views are views corresponding to N different viewpoints at the same time. For example, the N views are views obtained by N viewpoints at time t, for example, images obtained by cameras at N viewpoints capturing different angles of the same environment at time t.

In some embodiments, the N views are the first image of the k-th GOP of each viewpoint among the N viewpoints, where k is a positive integer.

For example, assume that N is 3. At this time, the three viewpoints are recorded as the first viewpoint, the second viewpoint and the third viewpoint, and the three views are recorded as the first viewpoint, the second viewpoint and the third viewpoint. Assume that the video data of the first viewpoint, the video data of the second viewpoint and the video data of the third viewpoint each include 100 images. When encoding the video data at each viewpoint, the video data at the viewpoint is first divided into multiple GOP groups. Assume that the 100 images included in the video data of the first viewpoint are divided into 5 groups, recorded as GOP11, GOP12, GOP13, GOP14, GOP15, and each group includes 20 images. Similarly, the 100 images included in the video data of the second viewpoint are divided into 5 groups, recorded as GOP21, GOP22, GOP23, GOP24, GOP25, and each group includes 20 images. The 100 images included in the video data of the third viewpoint are divided into 5 groups, which are recorded as GOP31, GOP32, GOP33, GOP34, and GOP35, and each group includes 20 images. At this time, the above three views may include the first view in GOP11, the first view in GOP21, and the first view in GOP31. Alternatively, the above three views may include the first view in GOP12, the first view in GOP22, and the first view in GOP32. Alternatively, the above three views may include the first view in GOP13, the first view in GOP23, and the first view in GOP33. Alternatively, the above three views may include the first view in GOP14, the first view in GOP24, and the first view in GOP34. Alternatively, the above three views may include the first view in GOP15, the first view in GOP25, and the first view in GOP35.

That is to say, in the embodiment of the present application, the decoding end only determines the color deviation fitting function of the first image in each GOP of N different viewpoint videos, and other views in the GOP reuse the color deviation fitting function of the first image in the GOP, which can reduce the number of color deviation fitting functions to be carried by the bitstream. number, thereby saving code words.

The decoding end decodes the N views. The decoding process of any non-basic view in the N views is basically the same. For ease of description, the decoding process of the i-th view is taken as an example for explanation, where i is a positive integer less than or equal to N.

For the i-th view, the decoding end first decodes the bitstream to obtain the patch information of the i-th view. In some embodiments, the patch information is also called patch data.

Next, the decoding end generates a patch image of the i-th view based on the patch information. For example, the decoding end maps each patch of the i-th view to a blank image using the position information of each patch included in the patch information of the i-th view to obtain a patch image of the i-th view.

Optionally, the size of the blank image is consistent with the size of the i-th view, that is, the patch image of the i-th view is consistent with the size of the view.

In an example, the i-th view includes 3 patches, and the 3 patches are mapped to the blank image to obtain a patch image as shown in FIG. 9 .

It can be seen from the above-mentioned TIMV14 encoding process that the multi-viewpoint video is grouped and each group is encoded separately. The above-mentioned N views can be understood as a group of views to be encoded. From the above, it can be seen that when N views are encoded, a basic view is selected from the N views, and the other views in the N views are compared with the basic view to eliminate redundant information, and the patch information after the redundant information is eliminated is encoded. The basic view is fully encoded. In this way, the decoding end can directly obtain the basic view by decoding the code stream, and then restore the other views based on the view and the patch information of other views. It can be seen that in an embodiment of the present application, the above-mentioned i-th view is a non-basic view among the N views.

S102: Divide the patch image into M image blocks according to a preset block division method.

Wherein, M is a positive integer.

The embodiment of the present application does not limit the specific method of pre-setting block division.

In some embodiments, the preset block division method may be to divide the patch image into M image blocks on average.

In some embodiments, the patch image is divided into M image blocks according to a preset block size.

In an example, as shown in FIG. 10 , the patch image shown in FIG. 9 is evenly divided into 4 image blocks, and each image block is processed separately. In this case, M is equal to 4.

In some embodiments, the encoding end indicates whether to turn on the color deviation fitting tool through a first flag. Therefore, before the decoding end divides the patch image into M image blocks according to a preset block division method, that is, before executing the above S102, it is necessary to first decode the code stream to obtain the first flag, and then determine whether to execute the above S102 based on the first flag. Specifically, if the first flag indicates that the decoding end turns on the color deviation fitting tool, it means that the code stream may include a color deviation fitting function. At this time, the decoding end divides the patch image into M image blocks according to the preset block division method, and executes the following step S103.

If the first flag indicates that the decoding end turns on the color deviation fitting tool, it means that the code stream does not include the color deviation fitting function. At this time, the decoding end skips the above step S102 and does not execute the following steps S103 and S104.

The embodiment of the present application does not limit the specific form of the first mark, as long as it is any information that can indicate whether to enable the color deviation fitting tool.

In one example, the field asme_cdpu_enabled_flag is used to represent the first flag. For example, if asme_cdpu_enabled_flag is a first value (such as 0), it indicates that the decoder enables the color deviation fitting tool. For another example, if asme_cdpu_enabled_flag is a second value (such as 1), it indicates that the decoder does not enable the color deviation fitting tool.

The embodiment of the present application does not limit the specific carrying position of the first flag in the code stream.

In a possible implementation manner, the first flag asme_cdpu_enabled_flag is carried in the atlas sequence parameter set MIV extended syntax.

Exemplarily, the atlas sequence parameter set MIV extended syntax is shown in Table 2:

Table 2

In an example, the first flag asme_cdpu_enabled_flag can also be used to indicate whether the code stream includes a color shift fitting function. For example, the first flag asme_cdpu_enabled_flag is 1, indicating that the code stream may include a color shift function pdu_cdpu_params. For example, the first flag asme_cdpu_enabled_flag is 0, indicating that the code stream does not include a color shift function pdu_cdpu_params. When asme_cdpu_enabled_flag does not exist, its value defaults to 0.

S103 . For the j-th image block among the M image blocks, decode the bitstream to obtain P color deviation fitting functions of the j-th image block.

Among them, the color deviation fitting function is obtained by performing chromaticity fitting on the j-th image block corresponding to the j-th image block, the image block is obtained by dividing the first pruning mask map into blocks according to a preset block division method, the first pruning mask map is obtained by pixel pruning of the i-th view, and j is a positive integer less than or equal to M.

In an embodiment of the present application, when encoding the i-th view among N views, the encoding end first determines the first pruning mask map of the i-th view, that is, adopts a pixel pruning method to determine the pixels that can be pruned in the i-th view, and sets the mask value (i.e., mask value) of the pixels that can be pruned to a value of 1 (e.g., 0). And determine the pixels that cannot be pruned in the i-th view, set the mask value (i.e., mask value) of the pixels that cannot be pruned in the i-th view to a value of 2 (e.g., 1), and then obtain the first pruning mask map of the i-th view. Then, the encoding end divides the i-th view into M image blocks according to a preset block division method. Then, the encoder performs chromaticity fitting on each image block in the M image blocks, for example, the jth image block, specifically, determines the unpruned pixels in the jth image block based on the first pruning mask image, and then performs chromaticity fitting on the unpruned pixels in the jth image block to obtain P color deviation fitting functions of the jth image block. Then, the encoder writes the P color deviation fitting functions of the jth image block into the bitstream.

It should be noted that the way in which the decoding end divides the patch image of the i-th view into blocks to obtain M image blocks is the same as the way in which the encoding end divides the i-th view into blocks to obtain M image blocks.

Based on this, the decoding end can obtain the color deviation fitting function of the j-th image block by decoding the bit stream, and then determine the color deviation fitting function as the color deviation fitting function of the j-th image block.

The embodiment of the present application does not limit the specific carrying position of the color shift function in the bit stream.

In a possible implementation, the color deviation fitting function is carried in a patch data unit. In this case, the decoding end can obtain the color deviation fitting function of the j-th image block by decoding the patch data unit.

In an example, the patch data unit includes a patch data unit MIV extended syntax, and the color shift fitting function is carried in the patch data unit MIV extended syntax.

Exemplarily, the patch data unit MIV extended syntax is shown in Table 3:

Table 3

In the above Table 3, pdu_cdpu_params[tileID][p] is the parameter of the color deviation fitting function of the j-th image block. When asme_cdpu_enable_flag does not exist, the pdu_cdpu_params value defaults to an identity transformation matrix, but this value will not be stored in the bitstream.

In some embodiments, if the first flag indicates that the decoding end starts the color deviation fitting tool, and the decoding end does not decode P color deviation fitting functions of the j-th image block in the bitstream, that is, the j-th image block cannot enable the color deviation fitting function to be reconstructed, then the method for the decoding end to reconstruct the j-th image block includes the following steps 1 and 2:

Step 1: Determine synthetic views of P parent nodes based on the P reconstructed images, and determine P synthetic view blocks of the j-th image block in the synthetic views of the P parent nodes;

Step 2: Based on the P synthetic view blocks, obtain the reconstruction value of the j-th image block.

Specifically, for each of the P parent nodes, the decoding end projects the reconstructed image of the parent node into the i-th viewport to obtain a synthetic view of the parent node. The corresponding image block of the j-th image block is determined in the synthetic view and recorded as a synthetic view block. In this way, for the P parent nodes, P synthetic view blocks can be obtained. Then, based on the P synthetic view blocks, a reconstruction value of the j-th image block is obtained. For example, the P synthetic view blocks are weighted to obtain a reconstruction value of the j-th image block.

In some embodiments, the decoding end divides the patch image of the i-th view into blocks, and the obtained j-th image block includes the following cases:

Case 1: All pixels in the j-th image block are not cropped.

Case 2: some pixels in the j-th image block are cropped, and some pixels are not cropped.

Case 3: All pixels in the j-th image block are cropped.

Based on the above three situations, for the j-th image block in situation 1, the j-th image block does not include the color deviation fitting function. Based on this, before executing the above S103, the decoding end first determines whether the j-th image block includes pruned pixels. If the decoding end determines that the j-th image block includes pruned pixels, the above S103 step is executed to decode the bitstream and obtain the color deviation fitting function corresponding to the j-th image block. If the j-th image block does not include pruned pixels, the decoding end skips the above S103 step and uses a related method, such as the above steps 1 and 2, to decode the j-th image block.

In some embodiments, for the j-th image block in situation 2, if the j-th image block includes fewer uncropped pixels, the encoding end, based on the consideration of encoding cost, does not determine the color deviation fitting function of the j-th image block. Based on this, before executing the above S103, the decoding end first determines whether the number of pruned pixels included in the j-th image block is greater than or equal to the preset value. If the decoding end determines that the number of pruned pixels included in the j-th image block is greater than or equal to the preset value, the above step S103 is executed to decode the bitstream and obtain the color deviation fitting function of the j-th image block. If the number of pruned pixels included in the j-th image block is less than the preset value, the decoding end skips the above step S103 and uses a related method, such as the above steps 1 and 2, to decode the j-th image block.

After the decoding end determines the color cast fitting function of the j-th image block based on the above steps, it executes the following step S104.

S104 . Use P color deviation fitting functions to perform pixel fitting on the cropped pixels in the j th image block to obtain a reconstructed block of the j th image block.

In an embodiment of the present application, after the decoding end obtains the color deviation fitting function of the j-th image block based on the above steps, it uses the color deviation fitting function to perform pixel fitting on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block.

In the embodiment of the present application, the decoding end uses a color deviation fitting function to perform pixel fitting on the cropped pixels in the j-th image block, and the specific method of obtaining the reconstructed block of the j-th image block is not limited.

In some embodiments, if the color deviation fitting function is obtained by fitting the image block corresponding to the j-th image block in the basic view among N views by the encoder, the decoder can use the color deviation fitting function to perform pixel fitting on the image block corresponding to the j-th image block in the basic view among N views to obtain a reconstructed block of the j-th image block.

In some embodiments, the above S104 includes the following steps S104-A to S104-C:

S104-A, determining a directed acyclic graph of pruning levels of N views;

S104-B, based on the directed acyclic graph, determining P reconstructed images corresponding to P parent nodes of the i-th view, where P is a positive integer;

S104-C: Based on the P reconstructed images, use the color deviation fitting function to perform pixel fitting on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block.

In this embodiment, the decoding end first determines a directed acyclic graph of the pruning hierarchy of N views, each node in the directed acyclic graph corresponds to one of the N views, and the view corresponding to the child node in the directed acyclic graph refers to the view of each parent node of the child node when pruning. In this way, when the decoding end decodes the i-th view among the N views, it can determine the parent node of the i-th view from the directed acyclic graph determined above. In the directed acyclic graph of the embodiment, the parent node of the i-th view may include one or more, which are recorded as P parent nodes for ease of description, where P is a positive integer. It should be noted that since the decoding end decodes from the root node to the child node based on the directed acyclic graph when decoding N views, when decoding the i-th view, the views on the P parent nodes of the i-th view have been decoded or reconstructed. In this way, when decoding the i-th view, the decoding end can directly obtain the reconstructed image of the view corresponding to the P parent nodes of the i-th view, that is, obtain P reconstructed images. In this way, the decoding end can use the color deviation fitting function based on the P reconstructed images to obtain the reconstructed block of the j-th image block for the cropped pixels in the j-th image block, thereby realizing accurate reconstruction of the j-th image block.

The specific process of determining the directed acyclic graph of the pruning levels of N views at the decoding end is described below:

The embodiment of the present application does not limit the specific manner in which the decoding end determines the directed acyclic graph of the pruning levels of N views.

In a possible implementation, the decoding end and the encoding end determine the default directed acyclic graph as a directed acyclic graph of the pruning levels of N views.

In a possible implementation, the encoder determines a directed acyclic graph of the pruning levels of N views, and indicates the directed acyclic graph of the pruning levels of the N views to the decoder. For example, the encoder writes the relevant information of the determined directed acyclic graph of the pruning levels of the N views into the bitstream. In this way, the decoder can obtain the directed acyclic graph of the pruning levels of the N views by decoding the bitstream.

For example, the directed acyclic graph of the pruning hierarchy of N views determined by the encoder is shown in FIG11. The encoder can write information that view V4 is a basic view into the bitstream, and information that view V1 is encoded based on view V4, information that view V3 is encoded based on view V4 and view V1, and information that view V0 is encoded based on view V4, view V1, and view V3 into the bitstream. In this way, the decoder can determine the reference relationship between views V0, V1, V3, and V4 by decoding the bitstream, and then restore the directed acyclic graph shown in FIG11 based on the reference relationship.

After the decoder determines the directed acyclic graph of the pruning levels of the N views based on the above steps, it executes the above step S104-B to determine the P reconstructed images corresponding to the P parent nodes of the i-th view based on the directed acyclic graph.

Continuing with the directed acyclic graph shown in FIG. 11 above, for example, when the i-th view is view V1, the parent node of view V1 is view V4, that is, the base view, that is, view V1 refers to base view V4 during decoding. For another example, if the i-th view is view V3, the parent nodes of view V3 include two, view V4 and view V1, that is, view V3 refers to base view V4 and view V1 during decoding. For another example, if the i-th view is view V0, the parent nodes of view V0 include three, view V4, view V1, and view V3, that is, view V0 refers to view V4, view V1, and view V3 during decoding.

Based on the above steps, after the decoder determines the P parent nodes of the i-th view, it can obtain the reconstructed image corresponding to each of the P parent nodes. Since one parent node corresponds to one view, P reconstructed images can be obtained for the P parent nodes.

After the decoding end determines the P reconstructed images corresponding to the i-th view, based on the P reconstructed images, the color deviation fitting function obtained by the above decoding is used to perform pixel fitting on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block.

In the embodiment of the present application, the decoding end performs pixel fitting on the cropped pixels in the j-th image block based on the P reconstructed images using the color deviation fitting function obtained by the above decoding to obtain the specific method of reconstructing the j-th image block.

In some embodiments, the decoding end uses a color deviation fitting function to perform pixel fitting on the P reconstructed images to obtain a fitting image, and determines the image block corresponding to the j-th image block in the fitting image as the reconstructed block of the j-th image block.

In some embodiments, the above S104-C includes the following steps S104-C1 to S104-C3:

S104-C1, determining synthetic views of P parent nodes based on the P reconstructed images, and determining P synthetic view blocks of the j-th image block in the synthetic views of the P parent nodes;

S104-C2, based on the P synthetic view blocks, using P color deviation fitting functions, fitting the pruned pixels in the j-th image block to obtain P color deviation reconstruction blocks of the j-th image block;

S104-C3, weighted fusion is performed on the P color-shifted reconstruction blocks to obtain a reconstruction block of the j-th image block.

In this embodiment, for each of the P parent nodes, the decoding end projects the reconstructed image of the parent node onto the i-th viewport to obtain a synthetic view of the parent node, and determines the corresponding image block of the j-th image block on the synthetic view, which is recorded as a synthetic view block. In this way, the P parent nodes can obtain P synthetic view blocks. For example, in the synthetic view of the first parent node among the P parent nodes, a synthetic view block of the j-th image block is determined, and in the synthetic view of the second parent node among the P parent nodes, a synthetic view block of the j-th image block is determined, and so on, P synthetic view blocks can be obtained.

Based on the above steps, the decoding end determines the P synthetic view blocks of the j-th image block in the P synthetic views, and then executes the above steps S104-C2, and fits the pruned pixels in the j-th image block based on the P synthetic view blocks using P color deviation fitting functions to obtain a reconstructed block of the j-th image block.

The embodiment of the present application does not limit the specific manner in which the decoding end fits the pruned pixels in the j-th image block based on P synthetic view blocks using P color deviation fitting functions to obtain the reconstructed block of the j-th image block.

In some embodiments, with an image block as a fitting unit, P color deviation fitting functions are used to fit P synthetic view blocks respectively, to obtain P color deviation reconstruction blocks of the j-th image block. For example, for synthetic view block 1 among the P synthetic view blocks, the color deviation fitting function corresponding to the synthetic view block 1 is used to fit synthetic view block 1, to obtain color deviation reconstruction block 1 of the j-th image block. For synthetic view block 2 among the P synthetic view blocks, the color deviation fitting function corresponding to the synthetic view block 2 is used to fit synthetic view block 2, to obtain color deviation reconstruction block 2 of the j-th image block. By analogy, P color deviation reconstruction blocks of the j-th image block can be obtained. Then, these P color deviation reconstruction blocks are weighted fused to obtain a fused reconstruction block. From the fused reconstructed block, the pixel values of the cropped pixels in the jth image block are determined. For example, for the cropped pixel 1 in the jth image block, the reconstruction value of the pixel corresponding to the pixel 1 in the fused reconstructed block is determined as the reconstruction value of the pixel 1, so that the reconstruction value of each cropped pixel in the jth image block can be determined. Then, the reconstruction values of the cropped pixels in the jth image block and the patch values of the non-cropped pixels in the jth image block are combined to obtain the reconstructed block of the jth image block.

In some embodiments, the decoding end restores the cropped pixels in the j-th image block pixel by pixel. In this case, the above S104-C2 includes the following steps S104-C21 to S104-C23:

S104-C21, for the k-th pruned pixel in the j-th image block, determine P synthetic pixels corresponding to the k-th pruned pixel in the P synthetic view blocks, where k is a positive integer;

S104-C22, based on the P synthetic pixels and the surrounding pixels of the P synthetic pixels, use the color deviation fitting function to perform pixel fitting on the k-th cropped pixel to obtain a reconstructed value of the k-th cropped pixel;

S104-C23, obtaining a reconstructed block of the j-th image block based on the reconstructed values of the pruned pixels in the j-th image block.

In this embodiment, the process of determining the reconstruction value of each cropped pixel in the jth image block by the decoding end is consistent. For ease of description, the kth cropped pixel in the jth image block is taken as an example for explanation.

In this embodiment, for the k-th pruned pixel in the j-th image block, the decoding end first determines the corresponding pixel of the k-th pruned pixel in each of the P synthesized image blocks corresponding to the j-th image block, and records them as synthesized pixel points. For example, in the first synthesized image block among the P synthesized image blocks, a corresponding pixel of the k-th pruned pixel is determined, and in the second synthesized image block among the P synthesized image blocks, a corresponding pixel of the k-th pruned pixel is determined, and so on, P corresponding pixel points of the k-th pruned pixel can be determined in the P synthesized image blocks, and recorded as P synthesized pixel points.

The P synthetic pixels have been reconstructed, and the decoding end can use P color deviation fitting functions to perform pixel fitting on the kth cropped pixel based on the reconstructed values of the P synthetic pixels to obtain the reconstructed value of the kth pruned pixel. For example, for each of the P synthetic pixels, the color deviation fitting function corresponding to the synthetic pixel is used to fit the reconstructed value of the synthetic pixel and the surrounding pixels of the synthetic pixel (for example, the pixel pattern in the 3X3 area) to obtain a color deviation fitting value. In this way, P color deviation fitting values can be obtained for the P synthetic pixels, and the P color deviation values are weighted and fused to determine the color deviation reconstruction value of the kth pruned pixel.

The embodiment of the present application does not limit the specific expression form of the color cast fitting function.

In one possible implementation, the color cast fitting function is a weighted least squares fitting function, so that the decoding end can perform weighted least squares fitting on the reconstructed value of the synthesized pixel point and the surrounding pixels of the synthesized pixel point to obtain the reconstructed value of the kth pruned pixel point.

Based on the above steps, the decoding end can determine the reconstruction value of each cropped pixel in the j-th image block, and then obtain the reconstruction value of the j-th image block.

The above embodiment introduces the process of determining the reconstruction value of the j-th image block in the patch image of the i-th view at the decoding end, and the reconstruction values of other image blocks in the patch image can be determined by referring to the determination process of the reconstruction value of the j-th image block, and then the reconstructed image of the i-th view can be obtained.

The above describes the decoding process of determining the i-th view among N views by the decoding end. The decoding end can decode other non-basic views among the N views with reference to the decoding process of the i-th view, thereby completing the decoding of the N views.

Exemplarily, as shown in FIG12, when encoding, the encoder merges the patch information of the two groups of views, wherein view V4 and view V6 are basic views, views V4, V1, V3, and V0 are decoded separately as a group of views, and views V6, V8, V7, and V5 are decoded separately as another group of views. When decoding the first group of views V4, V1, V3, and V0, the decoder first decodes the bitstream to obtain the patch information shown in FIG12, and obtains the patch information of the first group of views V4, V1, V3, and V0 from the patch information, wherein the basic view V4 is the basic view, and the entire information is encoded, so the basic view V4 can be reconstructed from the patch information. The decoder determines a directed acyclic graph of the pruning hierarchy of the first group of views V4, V1, V3, and V0, as shown in the upper left corner of FIG12. Based on the directed acyclic graph and the basic view V4, the view V1 is decoded, and specifically, based on the patch information of the view V1, a patch image of the view V1 is generated. Next, the patch image of view V1 is divided into M image blocks. For each of the M image blocks, for example, the j-th image block, the code stream is decoded to obtain the color deviation fitting function of the image block. Next, the cropped pixels in the j-th image block are pixel-fitted using the color deviation fitting function to obtain a reconstructed block of the j-th image block. The specific process refers to the description of the above embodiment. For example, based on the basic view V4, the color deviation fitting function is used to perform pixel fitting on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block. Repeating this process can achieve decoding of each of the M image blocks included in the patch image of view V1, thereby obtaining a decoded image of view V1. Next, based on the decoded images of the basic view V4 and view V1, view V3 is decoded to obtain a reconstructed image of view V3. Based on the decoded images of the basic view V4, view V1 and view V3, view V0 is decoded to obtain a reconstructed image of view V0.

For V6, V8, V7, and V5 in the second group of views, the same method can be used to decode and obtain the reconstructed images corresponding to V6, V8, V7, and V5 in the second group of views.

The video decoding method provided in the embodiment of the present application is that when the decoding end decodes the multi-view video, for the i-th view among N views, the decoding end decodes the bitstream, determines the patch information of the i-th view, and generates the patch image of the i-th view based on the patch information, where the N views are views of N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N; the patch image is divided into M image blocks according to a preset block division method, where M is a positive integer; for the j-th image block among the M image blocks, the decoding bitstream is performed to obtain P color deviation fitting functions of the j-th image block; using the P color deviation fitting functions, the cropped pixels in the j-th image block are pixel-fitted to obtain a reconstructed block of the j-th image block. That is, the embodiment of the present application performs fitting and reconstruction on the cropped pixels in the i-th view based on the color deviation fitting function, which can reduce the number of pixels that need to be encoded by the encoding end, thereby reducing the encoding cost of the encoding end and improving the decoding efficiency of the multi-view video.

The above describes the multi-view video decoding method of the present application by taking the decoding end as an example, and the following describes it by taking the encoding end as an example.

FIG13 is a schematic diagram of a video encoding method flow chart provided by an embodiment of the present application, and the embodiment of the present application is applied to the video encoders shown in FIG1 and FIG2. As shown in FIG13, the method of the embodiment of the present application includes:

S201 . For an i th view among N views, determine a first pruning mask map of the i th view.

At present, when encoding multi-viewpoint videos, the current pixel pruning method determines the pruned pixels, and does not prune the blocks with too large color difference, but retains them. For example, FIG. 14 shows four views taken from different perspectives. Due to camera parameters and other reasons, the color difference of the image blocks in the box taken by cameras from different viewpoints is large. The current pixel pruning method retains the area with large color difference, which will increase the encoding burden and encoding cost of the encoding end, thereby reducing the encoding efficiency.

In order to solve this technical problem, the embodiment of the present application performs chromaticity fitting function fitting on the area with large color difference, so that the encoding end does not need to retain the area with large color difference, but encodes the color deviation fitting function of the area into the bitstream. In this way, the decoding end can obtain the color deviation fitting function of the area by decoding the bitstream, and then obtain the area based on the color deviation fitting function. In this way, the encoding cost of the encoding end can be reduced while ensuring the decoding accuracy in advance, and the encoding efficiency of multi-viewpoint video can be improved.

In the embodiment of the present application, the N views are views from N different viewpoints, that is, the viewpoints corresponding to the N views are all different.

Wherein N is a positive integer greater than 1. That is, the N views in the embodiment of the present application include 2 views, 3 views, and any number of views greater than 2.

In some embodiments, the N views are views corresponding to N different viewpoints at the same time. For example, the N views are views obtained by N viewpoints at time t, for example, images obtained by cameras at N viewpoints capturing different angles of the same environment at time t.

In some embodiments, the N views are the first image of the k-th GOP of each viewpoint among the N viewpoints, where k is a positive integer.

For example, assume that N is 3, 3 viewpoints, respectively recorded as the first viewpoint, the second viewpoint and the third viewpoint, and 3 views, respectively recorded as the first viewpoint, the second viewpoint and the third viewpoint. Assume that the video data of the first viewpoint, the video data of the second viewpoint and the video data of the third viewpoint each include 100 images. When encoding the video data at each viewpoint, first divide the video data at the viewpoint into multiple GOP groups. Assume that the 100 images included in the video data of the first viewpoint are divided into 5 groups, recorded as GOP11, GOP12, GOP13, GOP14, GOP15, and each group includes 20 images. Similarly, the 100 images included in the video data of the second viewpoint are divided into 5 groups, recorded as GOP21, GOP22, GOP23, GOP24, GOP25, and each group includes 20 images. The 100 images included in the video data of the third viewpoint are divided into 5 groups, which are recorded as GOP31, GOP32, GOP33, GOP34, and GOP35, and each group includes 20 images. At this time, the above three views may include the first view in GOP11, the first view in GOP21, and the first view in GOP31. Alternatively, the above three views may include the first view in GOP12, the first view in GOP22, and the first view in GOP32. Alternatively, the above three views may include the first view in GOP13, the first view in GOP23, and the first view in GOP33. Alternatively, the above three views may include the first view in GOP14, the first view in GOP24, and the first view in GOP34. Alternatively, the above three views may include the first view in GOP15, the first view in GOP25, and the first view in GOP35.

That is to say, in an embodiment of the present application, the encoding end only determines the color deviation fitting function of the first image in each GOP of N different viewpoint videos, and other views in the GOP reuse the color deviation fitting function of the first image in the GOP. This can reduce the number of color deviation fitting functions to be carried by the code stream, thereby saving codewords.

It can be seen from the encoding process of TIMV14 mentioned above that the multi-viewpoint video is grouped and each group is encoded separately. The above N views can be understood as a group of views to be encoded. It can be seen from the above that when N views are encoded, a basic view is selected from the N views, and other views in the N views are compared with the basic view to eliminate redundant information, and the patch information after eliminating the redundant information is encoded. The basic view is fully encoded. In this way, the decoding end can directly obtain the basic view by decoding the code stream, and then restore other views based on the view and the patch information of other views. It can be seen that in the embodiment of the present application, the encoding process of other views except the basic view in the N views is mainly introduced. That is to say, the above i-th view is a non-basic view in the N views.

The encoding process of any non-basic view among the N views at the encoding end is basically the same. For ease of description, the encoding process of the i-th view is taken as an example for explanation, where i is a positive integer less than or equal to N.

For the i-th view, the encoding end first determines the first pruning mask map of the i-th view. The first pruning mask map is a pruning mask map obtained by performing pixel pruning on the i-th view. That is to say, the encoding end adopts a pixel pruning method to determine the pixels that can be pruned in the i-th view. If it is determined that the pixel can be pruned, the mask value (i.e., mask value) of the pixel that can be pruned is set to a value of 1 (e.g., 0). If it is determined that the pixel cannot be pruned, the mask value (i.e., mask value) of the pixel that cannot be pruned is set to a value of 2 (e.g., 1).

The specific process of determining the first pruning mask map of the i-th view at the encoder is introduced below.

The embodiment of the present application does not limit the specific method for the encoder to determine the first pruning mask map of the i-th view.

In some embodiments, the encoding end may project the i-th view into the base view, determine the pixels that can be hidden in the i-th view based on the difference (or similarity) between the i-th view and the base view, and then obtain the first pruning mask map of the i-th view. For example, for a certain pixel 1 in the i-th view, the encoding end projects the pixel 1 into the base view through the internal and external parameters of the camera and the depth information, assuming that the projection point of the pixel 1 in the base view is the pixel 2. By determining the similarity between the pixel 1 and the pixel 2, it is determined whether to crop the pixel 1. For example, if the similarity between the pixel 1 and the pixel 2 is greater than or equal to a preset value, it is determined that the pixel 1 can be cropped, and then the mask value corresponding to the pixel 1 is set to 0. If the similarity between the pixel 1 and the pixel 2 is less than a preset value, it is determined that the pixel 1 cannot be cropped, and then the mask value corresponding to the pixel 1 is set to 1. Based on this step, the first pruning mask map of the i-th view can be determined.

In some embodiments, the encoder may determine the first pruning mask image of the i-th view through the following steps S201-A to S201-C:

S201-A, generating a directed acyclic graph of pruning levels of N views;

S201-B, determining P synthetic views of P parent nodes of the i-th view based on the directed acyclic graph;

S201-C: Based on P synthetic views, perform pixel pruning on the i-th view to obtain a first pruning mask image.

In this embodiment, the encoding end first generates a directed acyclic graph of the pruning hierarchy of N views, each node in the directed acyclic graph corresponds to one of the N views, and the view corresponding to the child node in the directed acyclic graph refers to the view of each parent node of the child node when pruning. In this way, when the encoding end encodes the i-th view among the N views, it can determine the parent node of the i-th view from the directed acyclic graph determined above. In the directed acyclic graph of the embodiment of the present application, the parent node of the i-th view may include one or more, and for the convenience of description, it is recorded as P parent nodes, where P is a positive integer. It should be noted that since the encoding end encodes from the root node to the child node based on the directed acyclic graph when encoding the N views, therefore, when encoding the i-th view, the views on the P parent nodes of the i-th view are projected into the i-th view viewport respectively to obtain P synthetic views. In this way, the encoding end can perform pixel pruning on the i-th view based on the P synthetic views to obtain a first pruning mask map.

The specific process of determining the directed acyclic graph of the pruning levels of N views at the decoding end is described below:

In the embodiment of the present application, a basic view, such as N0, is determined from N views, and views other than the basic view in the N views are recorded as additional views, such as N1, N2, and N3. The following steps are included when creating a directed acyclic graph:

Step 1: Take the basic view N0 as the root node of the directed acyclic graph.

Step 2, project all the pixels of the basic view N0 into each additional view, determine the pruning mask map of each additional view, that is, the similarity between the pixel points in the additional view and the projected pixel points. If the similarity is large, the mask value of the pixel point is set to 0; if the similarity is small, the mask value of the pixel point is set to 1. Repeat this step to obtain the pruning mask map of the additional view.

Step 3: Based on the pruning mask images of each additional view, select the additional view with the smallest mask area, that is, the additional view with the largest number of retained pixels, from these additional views.

Step 4, make the selected accessory view as the child node of all nodes in the directed acyclic graph. If all views are assigned to nodes in the directed acyclic graph, stop, otherwise execute step 5.

Step 5: Project all the retained pixels (ie, the pixels that are not cropped) of the selected view to the remaining additional views.

Step 7, update the clipping mask for each remaining additional view.

Step 8, skip to step 4.

After the above steps, a directed acyclic graph as shown in FIG11 can be obtained.

In some embodiments, the encoder indicates the directed acyclic graph of the pruning hierarchy of N views to the decoder. For example, the directed acyclic graph of the pruning hierarchy of N views determined by the encoder is shown in FIG11. The encoder can write information that view V4 is a basic view into the bitstream, and information that view V1 is encoded based on view V4, information that view V3 is encoded based on view V4 and view V1, and information that view V0 is encoded based on view V4, view V1, and view V3 into the bitstream. In this way, the decoder can determine the reference relationship between views V0, V1, V3, and V4 by decoding the bitstream, and then restore the directed acyclic graph shown in FIG11 based on the reference relationship.

After the encoder generates a directed acyclic graph of the pruning hierarchy of N views based on the above steps, it executes the above step S201-B to determine P synthetic views of the P parent nodes of the i-th view based on the directed acyclic graph.

Continuing with the directed acyclic graph shown in FIG. 11 above, for example, when the i-th view is view V1, the parent node of view V1 is view V4, that is, the basic view, that is, view V1 refers to basic view V4 during encoding. For another example, if the i-th view is view V3, the parent nodes of view V3 include two, view V4 and view V1, that is, view V3 refers to basic view V4 and view V1 during encoding. For another example, if the i-th view is view V0, the parent nodes of view V0 include three, view V4, view V1, and view V3, that is, view V0 refers to view V4, view V1, and view V3 during encoding.

Based on the above steps, after the encoder determines the P parent nodes of the i-th view, it can determine the view of each parent node of the P parent nodes and project it into the i-th viewport to obtain the synthetic view of the parent node. Since one parent node corresponds to one synthetic view, P synthetic views can be obtained for P parent nodes.

After the encoder determines P synthetic views, it executes the above step S201-C to perform pixel pruning on the i-th view based on the P synthetic views to obtain a first pruning mask image.

The embodiment of the present application does not limit the specific manner in which the encoding end performs pixel pruning on the i-th view based on P synthetic views to obtain the first pruning mask map.

In some embodiments, the encoder determines the similarity between each pixel in the i-th view and the corresponding pixel in the P synthetic views, thereby obtaining a first pruning mask image.

In some embodiments, the above S201-C includes the following steps S201-C1 to S201-C2:

S201-C1, pruning the i-th view based on the difference between the brightness component of the i-th view and the brightness components of the P synthetic views, to obtain a second pruning mask map of the i-th view;

S201-C2: Based on the difference between the chroma components of the i-th view and the chroma components of the P synthetic views, query the non-pruned pixels in the pruned pixels included in the second pruning mask map to obtain a first pruning mask map.

In this embodiment, the pixel pruning module detects repeated parts between views so that the repeated pixels can be pruned in subsequent processing. As shown in FIG14B , an arbitrary pixel 1 of the i-th view is mapped to a pixel 2 of one of the P synthetic views and its 3x3 neighborhood through the internal and external parameters of the camera and the depth information. By detecting the similarity between pixel 1 and pixel 2, it is decided whether to prune pixel 1. Specifically, as shown in FIG14C , pixel pruning mainly includes two stages.

In the first stage of pruning pixels, two criteria are used to determine whether pairs of pixels are similar:

First, the depth value difference between pixel 1 and pixel 2 should be less than a first threshold t1. The depth value comparison is performed in a pixel-to-pixel manner.

Next, the minimum value of the brightness value difference between pixel 1 and all pixels in the 3×3 pixel block should be smaller than the second threshold t2, and the brightness value comparison is performed in a pixel-to-block manner.

After the first phase of detection, the suspected similar pixel pairs enter the second-pass process:

The second stage mainly updates the pruning mask created during the initial pruning stage and re-identifies the pixels not to be pruned among the pixels initially determined to be pruned. The main purpose of this process is to take into account the global color component differences that may exist between different views. The process is as follows and applies to each pruning pair. Specifically, for the pixels determined to be pruned, the pixel-by-pixel color difference between the P synthetic views and the i-th view is calculated. By using the least squares method, a fitting function that can best model these color differences is calculated. Pixels that meet the fitting function within a specific range defined by the threshold are judged as inner pixels, and these pixels are retained as pixels to be pruned. At the same time, outliers are updated to not be pruned within the pruning mask. After the second-pass process is completed, the similar pixel pairs to be finally cropped can be obtained, and then the first pruning mask map of the i-th view is obtained.

After the encoder determines the first pruning mask image of the i-th view based on the above steps, it performs the following step S202.

S202: Divide the i-th view into M image blocks based on a preset block division method.

Wherein, M is a positive integer.

The embodiment of the present application does not limit the specific method of pre-setting block division.

In some embodiments, the preset block division method may be to divide the i-th view into M image blocks on average.

In some embodiments, the i-th view is divided into M image blocks according to a preset block size.

In some embodiments, before executing the above S202, the encoder first determines a first flag, which is used to indicate whether the encoder turns on the color deviation fitting tool. If the first flag indicates that the encoder turns on the color deviation fitting tool, the encoder executes the above S202, and based on a preset block division method, the i-th view is divided into M image blocks.

In some embodiments, the encoding end writes the first flag into the bitstream.

For example, if the first flag indicates that the color cast fitting tool is turned on, the first flag is set to a first value (eg, 1).

For example, if the first flag indicates that the color cast fitting tool is not enabled, the first flag is set to a first value (eg, 1).

The embodiment of the present application does not limit the specific form of the first mark, as long as it is any information that can indicate whether to enable the color deviation fitting tool.

In one example, the field asme_cdpu_enabled_flag is used to represent the first flag. For example, if asme_cdpu_enabled_flag is a first value (such as 0), it indicates that the color cast fitting tool is enabled. For another example, if asme_cdpu_enabled_flag is a second value (such as 1), it indicates that the color cast fitting tool is not enabled.

The embodiment of the present application does not limit the specific carrying position of the first flag in the code stream.

In a possible implementation manner, the first flag asme_cdpu_enabled_flag is carried in the atlas sequence parameter set MIV extended syntax.

Exemplarily, the atlas sequence parameter set MIV extended syntax is shown in Table 2.

S203. For the j-th image block among the M image blocks, determine the unpruned pixels in the j-th image block based on the first pruning mask image, and perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain a color cast fitting function of the j-th image block.

Wherein, j is a positive integer less than or equal to M.

In the embodiment of the present application, the encoding end obtains the first pruning mask image based on the pixel pruning method, so that the unpruned pixels in each of the M image blocks of the i-th view can be determined based on the first pruning mask image, and then the unpruned pixels in each image block are pruned again to reduce the amount of coded data and reduce the coding cost. Specifically, the encoding end determines the color deviation fitting function of each of the M image blocks, and based on the color deviation fitting function, the unpruned pixels in the image block are pruned again to further reduce the coding cost.

In the embodiment of the present application, the method for the encoding end to determine the chromaticity fitting function of each of the M image blocks is basically the same. For the sake of ease of description, the j-th image block is again taken as an example for explanation.

The embodiment of the present application does not limit the specific manner in which the encoding end determines the color cast fitting function of the j-th image block.

In some embodiments, the encoding end determines the corresponding pixel points of the unpruned pixel points in the j-th image block in the basic view, determines P color deviation fitting functions based on these corresponding pixel points and the unpruned pixel points in the j-th image block, and determines the P color deviation fitting functions as the color deviation fitting functions of the j-th image block.

In some embodiments, the above S203 includes the following steps S203-A to S203-B:

S203-A, determining P synthetic view blocks of the j-th image block in P synthetic views;

S203-B: Based on the chromaticity values of the pixels included in the P synthetic view blocks and the chromaticity values of the pixels included in the j-th image block, perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain P color deviation fitting functions.

In this embodiment, the encoding end determines P synthetic views of the i-th view based on the above steps, so that P color deviation fitting functions of the j-th image block can be determined based on the P synthetic views.

Specifically, the encoder first determines an image block corresponding to the j-th image block in each of the P synthetic views determined above, and records it as a synthetic view block, and then obtains P synthetic view blocks. For example, in the first synthetic view of the P synthetic views, a synthetic view block of the j-th image block is determined, and in the second synthetic view of the P synthetic views, a synthetic view block of the j-th image block is determined, and so on, and P synthetic view blocks can be obtained.

Based on the above steps, the encoding end determines the P synthetic view blocks of the j-th image block in the P synthetic views, and then executes the above step S203-B to perform chromaticity fitting on the unpruned pixels in the j-th image block based on the chromaticity values of the pixels included in the P synthetic view blocks and the chromaticity values of the pixels included in the j-th image block, so as to obtain P color deviation fitting functions.

The embodiment of the present application does not limit the specific method of obtaining P color deviation fitting functions by performing chromaticity fitting on the unpruned pixels in the j-th image block based on the chromaticity values of the pixels included in the P synthetic view blocks and the chromaticity values of the pixels included in the j-th image block at the encoding end.

In some embodiments, for each of the P synthetic view blocks, the encoder performs least squares fitting (eg, weighted minimum multiplication fitting) based on the chrominance value of the synthetic view block and the chrominance value of the j-th image block to obtain a color deviation fitting function.

In some embodiments, for any one of the P synthetic view blocks, the encoder determines, in the synthetic view block, a synthetic pixel corresponding to an uncropped pixel in the j-th image block; and determines a color deviation fitting function based on the chromaticity value of the synthetic pixel and the chromaticity value of the uncropped pixel in the j-th image block. For example, for each uncropped pixel in the j-th image block, such as pixel 1, a synthetic pixel 2 corresponding to the pixel 1 is determined in the synthetic view block, and chromaticity fitting is performed on the chromaticity value of the pixel 1 based on the chromaticity values of the synthetic pixel 2 and the pixels around the synthetic pixel 2 (such as the pixels in a 3X3 area). Based on this method, a color deviation fitting function can be obtained by fitting the uncropped pixels in the j-th image block.

The embodiment of the present application does not limit the specific fitting method used in the color shift fitting process.

In one example, the encoding end performs weighted least squares fitting based on the chromaticity values of the synthesized pixel points and the chromaticity values of the uncropped pixel points in the j-th image block to obtain a color deviation fitting function.

In some embodiments, before the encoding end performs chromaticity fitting on the unpruned pixels in the jth image block to obtain the color deviation fitting function of the jth image block, the embodiment of the present application also includes: determining whether the jth image block includes unpruned pixels; if the jth image block includes unpruned pixels, performing chromaticity fitting on the unpruned pixels in the jth image block to obtain the color deviation fitting function.

In some embodiments, if the j-th image block includes untrimmed pixels, chromaticity fitting is performed on the untrimmed pixels in the j-th image block to obtain a color deviation fitting function, including: if the number of untrimmed pixels included in the j-th image block is greater than or equal to a first preset value, chromaticity fitting is performed on the untrimmed pixels in the j-th image block to obtain a color deviation fitting function.

After the encoder determines P color deviation fitting functions of the j-th image block based on the above steps, it executes the following step S204.

S204 , prune the unpruned pixels in the j th image block based on P color cast fitting functions to obtain patch information of the ith view.

In an embodiment of the present application, the encoding end determines P color deviation fitting functions of the j-th image block based on the above steps, and uses the P color deviation fitting functions to prune the unpruned pixels in the j-th image block to further reduce the encoding cost of the encoding end.

The embodiment of the present application does not limit the specific method of trimming the untrimmed pixels in the j-th image block based on P color cast fitting functions to obtain the patch information of the i-th view.

In some embodiments, for each un-pruned pixel in the j-th image block, P color deviation fitting functions are used to determine the un-pruned pixel. The pixel value of the pixel point is fitted, and then the fitting error is determined based on the comparison between the fitted pixel value and the original pixel value of the pixel point. If the fitting error is less than a preset value, the unpruned pixel point is determined as a pixel point that can be pruned.

In some embodiments, before the encoder uses P color deviation fitting functions to prune the unpruned pixels in the j-th image block, it is necessary to first determine whether the color deviation fitting function is valid. That is, it is determined whether the fitting error of the color deviation fitting function is less than or equal to the second preset value. If the fitting errors of the P color deviation fitting functions are less than or equal to the second preset value, it means that the P color deviation fitting functions are valid, and the P color deviation fitting functions are used to prune the unpruned pixels in the j-th image block. If the P color deviation fitting functions are invalid, the encoder uses the patch information of the i-th view based on the first pruning mask map.

The specific process of determining the fitting errors of the P color deviation fitting functions is introduced below.

In the embodiment of the present application, the encoding end can determine the fitting errors of P color deviation fitting functions through the following steps 3 and 4.

Step 3: for the kth untrimmed pixel in the jth image block, perform pixel fitting on the kth untrimmed pixel based on P color deviation fitting functions to determine the fitting error of the kth untrimmed pixel, where k is a positive integer;

Step 4: Determine the fitting errors of P color deviation fitting functions based on the fitting errors of the unpruned pixels in the j-th image block.

In this embodiment, the encoder first determines the fitting error of each unpruned pixel in the jth image block based on P color deviation fitting functions. The process of determining the fitting error of each unpruned pixel by the encoder is basically the same. For ease of description, the kth unpruned pixel is taken as an example for description.

The embodiment of the present application does not limit the specific method of performing pixel fitting on the kth unpruned pixel based on P color deviation fitting functions and determining the fitting error of the kth unpruned pixel.

In some embodiments, pixel fitting is performed on the kth untrimmed pixel based on P color deviation fitting functions to obtain a fitting pixel value of the kth untrimmed pixel, and a fitting error of the kth untrimmed pixel is determined based on the fitting pixel value of the kth untrimmed pixel and the original pixel value of the kth untrimmed pixel. The pixel value includes a YUV value.

In some embodiments, in the above step 3, performing pixel fitting on the kth unpruned pixel based on the P color deviation fitting functions and determining the fitting error of the kth unpruned pixel comprises the following steps:

Step 31: Use P color deviation fitting functions to perform chromaticity fitting on the kth unpruned pixel to obtain a fitting chromaticity value of the kth unpruned pixel;

Step 32: Determine the fitting error of the kth unpruned pixel based on the fitted chromaticity value of the kth unpruned pixel and the chromaticity value of the kth unpruned pixel.

In this embodiment, the color deviation error is determined by using P color deviation fitting functions to perform chromaticity fitting on the kth untrimmed pixel to obtain the fitted chromaticity value of the kth untrimmed pixel, and determining the fitting error of the kth untrimmed pixel based on the fitted chromaticity value of the kth untrimmed pixel and the chromaticity value of the kth untrimmed pixel.

Based on the above steps, the encoder can determine the fitting error of each unpruned pixel in the j-th image block, and then execute the above step 4.

In the embodiment of the present application, the implementation methods of the above step 4 include but are not limited to the following:

In method 1, the sum of the fitting errors of the unpruned pixels in the j-th image block is determined as the fitting error of the color deviation fitting function.

Method 2: The average value of the fitting errors of the unpruned pixels in the j-th image block is determined as the fitting error of the color deviation fitting function.

Based on the above steps, the encoder determines the fitting errors of the P color deviation fitting functions of the j-th image block. If the fitting errors of the P color deviation fitting functions are less than or equal to the second preset value, the P color deviation fitting functions are used to trim the untrimmed pixels in the j-th image block to obtain the patch information of the i-th view. If the P color deviation fitting functions are invalid, the encoder determines the patch information of the i-th view based on the first trimming mask map.

The specific process of pruning the unpruned pixels in the j-th image block based on the P color deviation fitting functions in the above S204 to obtain the patch information of the i-th view is introduced below.

In the embodiment of the present application, the specific method in which the encoder prunes the unpruned pixels in the j-th image block based on the color cast fitting function to obtain the patch information of the i-th view is not limited.

Based on the above steps, the encoder processes each of the M image blocks in the first pruning mask image and obtains the following situations:

Case 1: All M image blocks have valid color cast fitting functions.

Case 2: Some of the M image blocks have valid color cast fitting functions.

Case 3: None of the M image blocks has a valid color cast fitting function.

For situation 3, the encoding end adopts relevant technology for encoding, and the embodiment of the present application mainly encodes the above-mentioned situation 1 and situation 2.

In some embodiments, the encoding end prunes all unpruned pixels in an image block having a valid color deviation fitting function, and determines patch information of the i-th view based on the image block after all unpruned pixels in the image block having a valid color deviation fitting function are pruned.

In some embodiments, S204 includes the following steps S204-A and S204-B:

S204-A, trimming untrimmed pixels in the j-th image block based on P color deviation fitting functions to obtain a third trimming mask image of the j-th image block;

S204-B. Determine patch information of the i-th view based on the third cropping mask map of the image block in the i-th view.

In this embodiment, the encoding end performs pixel-by-pixel pruning on the unpruned pixels in the j-th image block based on the P color deviation fitting functions to obtain the third cropping mask map of the j-th image block. In this way, the third cropping mask map of the image block with the effective color deviation fitting function in the i-th view can be determined. The first cropping mask map is updated based on the third cropping mask map, and the patch information of the i-th view is determined based on the updated first cropping mask map.

The following describes a specific method of trimming the untrimmed pixels in the j-th image block based on the P color cast fitting functions in S204-A to obtain the third trimming mask image of the j-th image block.

In the embodiment of the present application, the specific method in which the encoding end trims the untrimmed pixels in the j-th image block based on P color deviation fitting functions to obtain the third cropping mask image of the j-th image block is not limited.

In some embodiments, the encoding end trims all untrimmed pixels in the j-th image block based on P color deviation fitting functions to obtain a third cropping mask image of the j-th image block.

In some embodiments, for each unpruned pixel in the j-th image block, for example, the k-th unpruned pixel, if the fitting error of the k-th unpruned pixel is less than or equal to a third preset value, the encoder cuts off the k-th unpruned pixel of the j-th image block to obtain a third cropping mask image of the j-th image block. If the fitting error of the k-th unpruned pixel is greater than the third preset value, the encoder does not trim the k-th unpruned pixel of the j-th image block to obtain a third cropping mask image of the j-th image block.

S205 : Encode the patch information and the color cast fitting function to obtain a bit stream.

Based on the above steps, the encoder can determine the patch information of the i-th view and the color deviation fitting function. Then, the encoder encodes the patch information and the color deviation fitting function to obtain a bitstream.

In some embodiments, if the first flag is set to a first value, the color deviation fitting function is encoded. That is, the encoding end writes the color deviation fitting function into the bitstream only when it determines that the color deviation fitting function of the image block is valid, that is, the fitting error of the color deviation fitting function is less than or equal to the second preset value.

In some embodiments, if the fitting error of the color deviation fitting function is greater than a second preset value, the encoding end skips the step of encoding the color deviation fitting function.

The embodiment of the present application does not limit the specific carrying position of the color shift function in the code stream.

In a possible implementation, the encoder writes the chromaticity function into the patch data unit. At this time, the decoder can obtain the color cast fitting function by decoding the patch data unit.

In an example, the patch data unit includes a patch data unit MIV extended syntax, and the color shift fitting function is carried in the patch data unit MIV extended syntax.

The above describes the encoding process of the i-th view among the N views by the encoder. The encoder can refer to the encoding process of the i-th view to encode other non-basic views among the N views, thereby completing the encoding of the N views.

Exemplarily, as shown in FIG. 15 and FIG. 16, when encoding, the encoder encodes two groups of views separately, wherein view V4 and view V6 are basic views, views V4, V1, V3, and V0 are individually encoded as a group of views, and views V6, V8, V7, and V5 are individually encoded as another group of views. When encoding the first group of views V4, V1, V3, and V0, the encoder first determines the first pruning mask map of the i-th view. Specifically, taking view V1 as an example, based on the basic view V4, view V1 is pixel-cropped, that is, for each pixel in view V1, it is determined whether the difference between the brightness value of the pixel and the brightness value of the projected pixel of the pixel in the basic view V4 is less than a preset value. If the difference between the brightness is greater than the preset value, the mask value of the pixel is set to 1. If the difference between the brightness is less than or equal to the preset value, the second stage of cropping is performed. Specifically, it is determined whether the deviation between the chromaticity value of the pixel and the chromaticity value of the projected pixel is greater than a preset value. If the deviation between the chromaticities is greater than a preset value, the mask value of the pixel is set to 1. If the deviation between the chromaticities is less than a preset value, the method of the embodiment of the present application is executed. Specifically, based on the above steps, the first pruning mask map of view V1 can be determined, and the i-th view is divided into M image blocks based on a preset block division method; for the j-th image block in the M image blocks, the unpruned pixels in the j-th image block are determined based on the first pruning mask map, and the unpruned pixels in the j-th image block are chromatically fitted to obtain a color deviation fitting function of the j-th image block; the unpruned pixels in the j-th image block are trimmed based on the color deviation fitting function to obtain patch information of the i-th view; the patch information and the color deviation fitting function are encoded to obtain a code stream. Based on the above steps, encoding of V3 and V0 in the first group of views can be achieved.

For V6, V8, V7, and V5 in the second group of views, the same method can be used to encode the patch information and color deviation fitting functions corresponding to V6, V8, V7, and V5 in the second group of views, and then encode the patch information and the color deviation fitting function to obtain a bitstream.

The video encoding method provided in the embodiment of the present application, when encoding a multi-view video, first determines the first pruning mask map of the i-th view in N views based on the pixel pruning method. Then, based on the preset block division method, the i-th view is divided into M image blocks, and for the j-th image block in the M image blocks, the unpruned pixels in the j-th image block are determined based on the first pruning mask map, and the unpruned pixels in the j-th image block are chromatically fitted to obtain the color deviation fitting function of the j-th image block. Then, the unpruned pixels in the j-th image block are pruned based on the color deviation fitting function to obtain the patch information of the i-th view. Finally, the above patch information and the color deviation fitting function are encoded to obtain a code stream. That is, the embodiment of the present application prunes the first pruning mask map obtained by pixel pruning again based on the color deviation fitting function to further reduce the number of unpruned pixels, reduce the amount of data required to be encoded at the encoding end, and thus reduce the encoding cost of the encoding end, and improve the encoding efficiency of the multi-view video.

It should be understood that Figures 8 to 16 are merely examples of the present application and should not be construed as limitations to the present application.

The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

It should also be understood that in the various method embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the embodiments of the present application, the term "and/or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. Specifically, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in the present application generally indicates that the associated objects before and after are in an "or" relationship.

The method embodiment of the present application is described in detail above in conjunction with Figures 8 to 16 , and the device embodiment of the present application is described in detail below in conjunction with Figures 10 to 13 .

FIG. 17 is a schematic block diagram of a video decoding device provided in an embodiment of the present application. The video decoding device 10 is applied to the above-mentioned video decoder.

As shown in FIG. 17 , the video decoding device 10 includes:

A determining unit 11 is used to decode a bitstream for an i-th view among N views, determine patch information of the i-th view, and generate a patch image of the i-th view based on the patch information, wherein the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N;

A dividing unit 12, configured to divide the patch image into M image blocks according to a preset block dividing method, where M is a positive integer;

A decoding unit 13 is used to decode the code stream for a j-th image block among the M image blocks to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer;

The fitting unit 14 is configured to perform pixel fitting on the cropped pixels in the j-th image block using the P color deviation fitting functions to obtain a reconstructed block of the j-th image block.

In some embodiments, the fitting unit 14 is specifically used to determine a directed acyclic graph of the pruning hierarchy of the N views; based on the directed acyclic graph, determine P reconstructed images corresponding to the P parent nodes of the i-th view, where P is a positive integer; based on the P reconstructed images, use the P color deviation fitting functions to perform pixel fitting on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block.

In some embodiments, the fitting unit 14 is specifically used to determine the synthetic views of the P parent nodes based on the P reconstructed images, and determine the P synthetic view blocks of the j-th image block in the synthetic views of the P parent nodes; based on the P synthetic view blocks, use the P color deviation fitting functions to fit the pruned pixels in the j-th image block to obtain the P color deviation reconstructed blocks of the j-th image block; and perform weighted fusion on the P color deviation reconstructed blocks to obtain the reconstructed block of the j-th image block.

In some embodiments, the fitting unit 14 is specifically used to determine, for the kth trimmed pixel in the jth second image block, the P synthetic pixels corresponding to the kth trimmed pixel in the P synthetic view blocks, where k is a positive integer; based on the P synthetic pixels and the surrounding pixels of the P synthetic pixels, use the color deviation fitting function to perform pixel fitting on the kth trimmed pixel to obtain a color deviation reconstruction value of the kth trimmed pixel; based on the color deviation reconstruction value of the trimmed pixel in the jth image block, obtain a color deviation reconstruction block of the jth image block.

In some embodiments, the decoding unit 13 is also used to decode the code stream to obtain a first flag before dividing the patch image into M image blocks according to a preset block division method, and the first flag is used to indicate whether to turn on the color deviation fitting tool; if the first flag indicates to turn on the color deviation fitting tool, the patch image is divided into M image blocks according to the preset block division method.

In some embodiments, the decoding unit 13 is further configured to skip the step of dividing the patch image into M image blocks according to a preset block division method if the first flag indicates that the color cast fitting tool is not enabled.

In some embodiments, if the first flag indicates that the color deviation fitting tool is turned on, and the P color deviation fitting functions of the j-th image block are not decoded in the code stream, the fitting unit 14 is further used to determine the synthetic views of the P parent nodes based on the P reconstructed images, and determine the P synthetic view blocks of the j-th image block in the synthetic views of the P parent nodes; based on the P synthetic view blocks, obtain the reconstructed block of the j-th image block.

In some embodiments, the fitting unit 14 is specifically configured to perform weighted processing on the P synthetic view blocks to obtain a reconstructed block of the j-th image block.

In some embodiments, the decoding unit 13 is specifically configured to decode the bitstream to obtain P color deviation fitting functions of the j-th image block if the j-th image block includes pruned pixels.

In some embodiments, the decoding unit 13 is specifically configured to decode the bitstream to obtain P color deviation fitting functions of the jth image block if the number of pruned pixels included in the jth image block is greater than or equal to a preset value.

In some embodiments, the N views are views corresponding to the N different viewpoints at the same time.

In some embodiments, the N views are the first image of the kth GOP of each viewpoint among the N viewpoints, and k is a positive integer.

In some embodiments, the fitting unit 14 is further configured to determine the color deviation fitting function corresponding to the i-th view as the color deviation fitting function of other views in the GOP where the i-th view is located.

In some embodiments, the code stream includes a patch data unit, and the decoding unit 13 is specifically configured to decode the patch data unit to obtain P color deviation fitting functions of the j-th image block.

In some embodiments, the i-th view is a non-base view among the N views.

It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device 10 shown in FIG. 17 can execute the decoding method of the decoding end of the embodiment of the present application, and the aforementioned and other operations and/or functions of each unit in the device 10 are respectively for implementing the corresponding processes in each method such as the decoding method of the decoding end, and for the sake of brevity, no further description is given here.

FIG18 is a schematic block diagram of a video encoding device provided in an embodiment of the present application, and the video encoding device is applied to the above-mentioned encoder.

As shown in FIG. 18 , the video encoding device 20 may include:

a determining unit 21, configured to determine, for an i-th view among N views, a first pruning mask map of the i-th view, wherein the first pruning mask map is a pruning mask map obtained by performing pixel pruning on the i-th view, wherein the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N;

A dividing unit 22, configured to divide the i-th view into M image blocks based on a preset block dividing method, where M is a positive integer;

a fitting unit 23, configured to determine, for a j-th image block among the M image blocks, unpruned pixels in the j-th image block based on the first pruning mask image, and perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer;

A pruning unit 24, configured to prune unpruned pixels in the j-th image block based on the P color cast fitting functions to obtain patch information of the i-th view;

The encoding unit 25 is used to encode the patch information and the color cast fitting function to obtain a code stream.

In some embodiments, the determination unit 21 is specifically used to generate a directed acyclic graph of the pruning hierarchy of the N views; based on the directed acyclic graph, determine P synthetic views of the P parent nodes of the i-th view; based on the P synthetic views, perform pixel pruning on the i-th view to obtain the first pruning mask map.

In some embodiments, the determination unit 21 is specifically used to prune the i-th view based on the difference between the luminance component of the i-th view and the luminance component of the P synthetic views to obtain a second pruning mask map of the i-th view; based on the difference between the chrominance component of the i-th view and the chrominance component of the P synthetic views, query the non-pruned pixels among the pruned pixels included in the second pruning mask map to obtain the first pruning mask map.

In some embodiments, the fitting unit 23, before performing chromaticity fitting on the untrimmed pixels in the j-th image block to obtain the color deviation fitting function of the j-th image block, is also used to determine whether the j-th image block includes untrimmed pixels; if the j-th image block includes untrimmed pixels, chromaticity fitting is performed on the untrimmed pixels in the j-th image block to obtain the P color deviation fitting functions.

In some embodiments, the fitting unit 23 is specifically used to perform chromaticity fitting on the untrimmed pixels in the jth image block if the number of untrimmed pixels included in the jth image block is greater than or equal to a first preset value to obtain the color deviation fitting function.

In some embodiments, the fitting unit 23 is specifically used to determine the P corresponding image blocks of the j-th image block in the P synthetic views; based on the chromaticity values of the pixels included in the P corresponding image blocks and the chromaticity values of the pixels included in the j-th image block, perform chromaticity fitting on the untrimmed pixels in the j-th image block to obtain the color deviation fitting function.

In some embodiments, the fitting unit 23 is specifically used to perform chromaticity fitting on the unpruned pixels in the jth image block if the number of unpruned pixels included in the jth image block is greater than or equal to a first preset value to obtain the P color deviation fitting functions.

In some embodiments, the fitting unit 23 is specifically used to determine P synthetic view blocks of the j-th image block in the P synthetic views; based on the chromaticity values of the pixels included in the P synthetic view blocks and the chromaticity values of the pixels included in the j-th image block, perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain the P color deviation fitting functions.

In some embodiments, the fitting unit 23 is specifically used to determine, for any one of the P synthetic view blocks, in the synthetic view block, a synthetic pixel point corresponding to an uncropped pixel point in the j-th image block; and determine the color deviation fitting function based on the chromaticity value of the synthetic pixel point and the chromaticity value of the uncropped pixel point in the j-th image block.

In some embodiments, the fitting unit 23 is specifically configured to perform weighted least squares fitting based on the chromaticity values of the synthesized pixel points and the chromaticity values of the uncropped pixel points in the j-th image block to obtain the color deviation fitting function.

In some embodiments, before the trimming unit 24 trims the untrimmed pixels in the jth image block based on the P color deviation fitting functions to obtain the patch information of the i-th view, it is also used to perform pixel fitting on the k-th untrimmed pixel in the j-th image block based on the P color deviation fitting functions to determine the fitting error of the k-th untrimmed pixel, where k is a positive integer; determine the fitting errors of the P color deviation fitting functions based on the fitting errors of the untrimmed pixels in the j-th image block; if the fitting errors of the P color deviation fitting functions are less than or equal to a second preset value, trim the untrimmed pixels in the j-th image block based on the color deviation fitting function to obtain the patch information of the i-th view.

In some embodiments, the trimming unit 24 is specifically used to trim the untrimmed pixels in the j-th image block based on the P color deviation fitting functions to obtain a third cropping mask image of the j-th image block; and determine the patch information of the i-th view based on the third cropping mask image of the image block in the i-th view.

In some embodiments, the pruning unit 24 is specifically used to use the P color deviation fitting functions to perform chromaticity fitting on the kth unpruned pixel to obtain the fitted chromaticity value of the kth unpruned pixel; based on the fitted chromaticity value of the kth unpruned pixel and the chromaticity value of the kth unpruned pixel, determine the fitting error of the kth unpruned pixel.

In some embodiments, the trimming unit 24 is specifically used to use the color deviation fitting function to perform chromaticity fitting on the kth untrimmed pixel to obtain the fitted chromaticity value of the kth untrimmed pixel; based on the fitted chromaticity value of the kth untrimmed pixel and the chromaticity value of the kth untrimmed pixel, determine the fitting error of the kth untrimmed pixel.

In some embodiments, the pruning unit 24 is specifically configured to determine the sum of the fitting errors of the unpruned pixels in the j-th image block as the fitting errors of the P color deviation fitting functions.

In some embodiments, the pruning unit 24 is specifically configured to determine an average value of fitting errors of unpruned pixels in the j-th image block as the fitting errors of the P color deviation fitting functions.

In some embodiments, the encoding unit 25 is further used to determine a first flag before dividing the i-th view into M image blocks based on a preset block division method, where the first flag is used to indicate whether to turn on the color cast fitting tool; if the first flag indicates to turn on the color cast fitting tool, the i-th view is divided into M image blocks based on the preset block division method.

In some embodiments, the encoding unit 25 is further configured to write the first flag into the bit stream.

In some embodiments, the encoding unit 25 is further configured to skip the step of encoding the color deviation fitting function if the fitting errors of the P color deviation fitting functions are greater than the second preset value.

In some embodiments, the N views are views generated by the N different viewpoints at the same time.

In some embodiments, the N views are the first image of the kth GOP of each viewpoint among the N viewpoints, and k is a positive integer.

In some embodiments, the fitting unit 24 is further configured to determine the color deviation fitting function corresponding to the i-th view as the color deviation fitting function of other views in the GOP where the i-th view is located.

In some embodiments, the encoding unit 25 is specifically configured to write the color cast fitting function into the patch data unit.

In some embodiments, the i-th view is a non-base view among the N views.

It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, it will not be repeated here. Specifically, the device 20 shown in Figure 18 may correspond to the corresponding subject in the encoding method of the encoding end of the embodiment of the present application, and the aforementioned and other operations and/or functions of each unit in the device 20 are respectively for implementing the corresponding processes in each method such as the encoding method of the encoding end, and for the sake of brevity, it will not be repeated here.

The above describes the device and system of the embodiment of the present application from the perspective of the functional unit in conjunction with the accompanying drawings. It should be understood that the functional unit can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software units. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and/or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software units in the decoding processor to perform. Optionally, the software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

FIG. 19 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

As shown in FIG. 19 , the electronic device 30 may be a video encoder or a video decoder as described in the embodiment of the present application, and the electronic device 30 may include:

The memory 33 and the processor 32, the memory 33 is used to store the computer program 34 and transmit the program code 34 to the processor 32. In other words, the processor 32 can call and run the computer program 34 from the memory 33 to implement the method in the embodiment of the present application.

For example, the processor 32 may be configured to execute the steps in the above method according to the instructions in the computer program 34 .

In some embodiments of the present application, the processor 32 may include but is not limited to:

General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

In some embodiments of the present application, the memory 33 includes but is not limited to:

Volatile memory and/or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct RAM bus random access memory (Direct Rambus RAM, DR RAM).

In some embodiments of the present application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to complete the method provided by the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 34 in the electronic device 30.

As shown in FIG. 19 , the electronic device 30 may further include:

The transceiver 33 may be connected to the processor 32 or the memory 33 .

The processor 32 may control the transceiver 33 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

FIG. 20 is a schematic block diagram of a video encoding and decoding system provided in an embodiment of the present application.

As shown in FIG. 20 , the video encoding and decoding system 40 may include: a video encoder 41 and a video decoder 42 , wherein the video encoder 41 is used to execute the video encoding method involved in the embodiment of the present application, and the video decoder 42 is used to execute the video decoding method involved in the embodiment of the present application.

The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

The present application also provides a code stream, which is generated according to the above encoding method.

When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (42)

A video decoding method, comprising: For an i-th view among N views, decode a bitstream, determine patch information of the i-th view, and generate a patch image of the i-th view based on the patch information, wherein the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N; According to a preset block division method, the patch image is divided into M image blocks, where M is a positive integer; For a j-th image block among the M image blocks, decode the bitstream to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer; Using the P color deviation fitting functions, pixel fitting is performed on the cropped pixels in the j-th image block to obtain a reconstructed block of the j-th image block. The method according to claim 1, characterized in that the step of using the P color deviation fitting functions to obtain a reconstructed block of the j-th image block for the cropped pixels in the j-th image block comprises: Determining a directed acyclic graph of pruning levels of the N views; Based on the directed acyclic graph, determine P reconstructed images corresponding to P parent nodes of the i-th view, where P is a positive integer; Based on the P reconstructed images, pixel fitting is performed on the cropped pixels in the j-th image block using the P color deviation fitting functions to obtain a reconstructed block of the j-th image block. The method according to claim 2, characterized in that the step of performing pixel fitting on the cropped pixels in the j-th image block based on the P reconstructed images using the P color deviation fitting functions to obtain a reconstructed block of the j-th image block comprises: Determine synthetic views of the P parent nodes based on the P reconstructed images, and determine P synthetic view blocks of the j-th image block in the synthetic views of the P parent nodes; Based on the P synthetic view blocks, using the P color deviation fitting functions, fitting the pruned pixels in the j-th image block to obtain P color deviation reconstruction blocks of the j-th image block; The P color-shifted reconstruction blocks are weightedly fused to obtain a reconstruction block of the j-th image block. The method according to claim 3, characterized in that the step of fitting the pruned pixels in the j-th image block using the P color deviation fitting functions based on the P synthetic view blocks to obtain the P color deviation reconstructed blocks of the j-th image block comprises: For a k-th pruned pixel in the j-th second image block, determine P synthetic pixels corresponding to the k-th pruned pixel in the P synthetic view blocks, where k is a positive integer; Based on the P synthetic pixels and the surrounding pixels of the P synthetic pixels, use the color deviation fitting function to perform pixel fitting on the k-th cropped pixel to obtain a color deviation reconstruction value of the k-th cropped pixel; Based on the color deviation reconstruction values of the pruned pixels in the j-th image block, a color deviation reconstruction block of the j-th image block is obtained. The method according to claim 2, characterized in that before dividing the patch image into M image blocks according to a preset block division method, the method further comprises: Decoding the bit stream to obtain a first flag, where the first flag is used to indicate whether to enable a color cast fitting tool; The step of dividing the patch image into M image blocks according to a preset block division method includes: If the first flag indicates to start the color cast fitting tool, the patch image is divided into M image blocks according to a preset block division method. The method according to claim 5, characterized in that the method further comprises: If the first flag indicates that the color cast fitting tool is not enabled, the step of dividing the patch image into M image blocks according to a preset block division method is skipped. The method according to claim 6, characterized in that if the first flag indicates to enable the color deviation fitting tool, and P color deviation fitting functions of the j-th image block are not decoded in the bitstream, the method further comprises: Determine synthetic views of the P parent nodes based on the P reconstructed images, and determine P synthetic view blocks of the j-th image block in the synthetic views of the P parent nodes; A reconstructed block of the j-th image block is obtained based on the P synthetic view blocks. The method according to claim 7, characterized in that the obtaining the reconstructed block of the j-th image block based on the P synthetic view blocks comprises: A weighted process is performed on the P synthetic view blocks to obtain a reconstructed block of the j-th image block. The method according to claim 2, characterized in that the decoding of the bit stream to obtain P color deviation fitting functions of the j-th image block comprises: If the j-th image block includes pruned pixels, the code stream is decoded to obtain P color deviation fitting functions of the j-th image block. The method according to claim 9, characterized in that if the j-th image block includes pruned pixels, decoding the bitstream to obtain P color deviation fitting functions of the j-th image block comprises: If the number of pruned pixels included in the j-th image block is greater than or equal to a preset value, the code stream is decoded to obtain P color deviation fitting functions of the j-th image block. The method according to claim 1 is characterized in that the N views are views corresponding to the N different viewpoints at the same time. The method according to claim 1 is characterized in that the N views are the first image of the kth GOP of each viewpoint among the N viewpoints, and k is a positive integer. The method according to claim 12, characterized in that the method further comprises: The color deviation fitting function of the i-th view is determined as the color deviation fitting functions of other views in the GOP where the i-th view is located. The method according to claim 1, characterized in that the bitstream includes a patch data unit, and the decoding of the bitstream to obtain P color deviation fitting functions of the j-th image block comprises: The patch data unit is decoded to obtain P color cast fitting functions of the j-th image block. The method according to claim 1 is characterized in that the i-th view is a non-basic view among the N views. A video encoding method, comprising: For an i-th view among N views, determine a first pruning mask map of the i-th view, where the first pruning mask map is a pruning mask map obtained by performing pixel pruning on the i-th view, where the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N; Based on a preset block division method, the i-th view is divided into M image blocks, where M is a positive integer; For a j-th image block among the M image blocks, determine unpruned pixels in the j-th image block based on the first pruning mask image, and perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer; Pruning unpruned pixels in the j-th image block based on the P color cast fitting functions to obtain patch information of the i-th view; The patch information and the color cast fitting function are encoded to obtain a bit stream. The method according to claim 16, characterized in that determining the first pruning mask map of the i-th view comprises: generating a directed acyclic graph of the pruning hierarchy of the N views; Determine, based on the directed acyclic graph, P synthetic views of the P parent nodes of the i-th view; Based on the P synthetic views, pixel pruning is performed on the i-th view to obtain the first pruning mask map. The method according to claim 17, characterized in that the step of performing pixel pruning on the i-th view based on the P synthetic views to obtain the first pruning mask map comprises: Based on a difference between a brightness component of the i-th view and brightness components of the P synthetic views, pruning the i-th view to obtain a second pruning mask map of the i-th view; Based on the difference between the chroma components of the i-th view and the chroma components of the P synthetic views, non-pruned pixels are queried among the pruned pixels included in the second pruning mask map to obtain the first pruning mask map. The method according to claim 17, characterized in that before performing chromaticity fitting on the unpruned pixels in the j-th image block to obtain the color deviation fitting function of the j-th image block, the method further comprises: Determining whether the j-th image block includes unpruned pixels; The performing chromaticity fitting on the unpruned pixel points in the j-th image block to obtain P color deviation fitting functions of the j-th image block includes: If the j-th image block includes unpruned pixels, chromaticity fitting is performed on the unpruned pixels in the j-th image block to obtain the P color deviation fitting functions. The method according to claim 19, characterized in that if the j-th image block includes unpruned pixels, performing chromaticity fitting on the unpruned pixels in the j-th image block to obtain the P color deviation fitting functions comprises: If the number of unpruned pixels included in the j-th image block is greater than or equal to a first preset value, chromaticity fitting is performed on the unpruned pixels in the j-th image block to obtain the P color deviation fitting functions. The method according to any one of claims 17 to 20, characterized in that the step of performing chromaticity fitting on the unpruned pixels in the j-th image block to obtain a color deviation fitting function of the j-th image block comprises: Determining P synthetic view blocks of the j-th image block among the P synthetic views; Based on the chromaticity values of the pixels included in the P synthetic view blocks and the chromaticity values of the pixels included in the j-th image block, chromaticity fitting is performed on the unpruned pixels in the j-th image block to obtain the P color deviation fitting functions. The method according to claim 21, characterized in that the performing chromaticity fitting on the unpruned pixels in the j-th image block based on the chromaticity values of the pixels included in the P synthetic view blocks and the chromaticity values of the pixels included in the j-th image block to obtain the P color deviation fitting functions comprises: For any synthetic view block among the P synthetic view blocks, determining, in the synthetic view block, a synthetic pixel point corresponding to an uncropped pixel point in the j-th image block; The color cast fitting function is determined based on the chromaticity value of the synthesized pixel and the chromaticity value of the uncropped pixel in the j-th image block. The method according to claim 22, characterized in that the determining the color cast fitting function based on the chromaticity value of the synthesized pixel and the chromaticity value of the uncropped pixel in the j-th image block comprises: Based on the chromaticity value of the synthesized pixel and the chromaticity value of the uncropped pixel in the j-th image block, a weighted least squares fitting is performed to obtain the color deviation fitting function. The method according to any one of claims 17 to 20, characterized in that before the step of trimming the untrimmed pixels in the j-th image block based on the P color cast fitting functions to obtain the patch information of the i-th view, the method further comprises: For a kth unpruned pixel in the jth image block, performing pixel fitting on the kth unpruned pixel based on the P color deviation fitting functions to determine a fitting error of the kth unpruned pixel, where k is a positive integer; Determining fitting errors of the P color deviation fitting functions based on fitting errors of unpruned pixels in the j-th image block; The step of trimming the untrimmed pixels in the j-th image block based on the P color cast fitting functions to obtain patch information of the i-th view includes: If the fitting errors of the P color deviation fitting functions are less than or equal to a second preset value, the unpruned pixels in the j-th image block are pruned based on the color deviation fitting function to obtain patch information of the i-th view. The method according to claim 24, characterized in that the step of trimming the untrimmed pixels in the j-th image block based on the P color deviation fitting functions to obtain the patch information of the i-th view comprises: The unpruned pixels in the j-th image block are pruned based on the P color deviation fitting functions to obtain a third pruned pixel of the j-th image block. Clip mask map; Based on a third cropping mask map of the image block in the i-th view, patch information of the i-th view is determined. The method according to claim 25, characterized in that the step of trimming the untrimmed pixels in the j-th image block based on the P color shift fitting functions to obtain a third cropping mask image of the j-th image block comprises: For the kth untrimmed pixel in the jth image block, if the fitting error of the kth untrimmed pixel is less than a third preset value, the kth untrimmed pixel of the jth image block is cropped to obtain a third cropping mask image of the jth image block. The method according to claim 24, characterized in that the performing pixel fitting on the kth unpruned pixel based on the P color deviation fitting functions to determine the fitting error of the kth unpruned pixel comprises: Using the P color deviation fitting functions, performing chromaticity fitting on the k-th unpruned pixel point to obtain a fitting chromaticity value of the k-th unpruned pixel point; A fitting error of the kth unpruned pixel is determined based on the fitted chromaticity value of the kth unpruned pixel and the chromaticity value of the kth unpruned pixel. The method according to claim 24, characterized in that the determining the fitting errors of the P color deviation fitting functions based on the fitting errors of the unpruned pixels in the j-th image block comprises: The sum of the fitting errors of the unpruned pixels in the j-th image block is determined as the fitting errors of the P color deviation fitting functions. The method according to claim 24, characterized in that the determining the fitting errors of the P color deviation fitting functions based on the fitting errors of the unpruned pixels in the j-th image block comprises: The average value of the fitting errors of the unpruned pixels in the j-th image block is determined as the fitting errors of the P color deviation fitting functions. The method according to claim 24, characterized in that before dividing the i-th view into M image blocks based on a preset block division method, the method further comprises: Determine a first flag, where the first flag is used to indicate whether to enable a color cast fitting tool; The step of dividing the i-th view into M image blocks based on a preset block division method includes: If the first flag indicates that the color cast fitting tool is enabled, the i-th view is divided into M image blocks based on a preset block division method. The method according to claim 30, characterized in that the method further comprises: The first flag is written into the bit stream. The method according to claim 24, characterized in that the method further comprises: If the fitting errors of the P color deviation fitting functions are greater than the second preset value, the step of encoding the color deviation fitting functions is skipped. The method according to any one of claims 16 to 20 is characterized in that the N views are views generated by the N different viewpoints at the same time. The method according to any one of claims 16 to 20, characterized in that the N views are the first image of the kth GOP of each viewpoint among the N viewpoints, and k is a positive integer. The method according to claim 34, characterized in that the method further comprises: The color deviation fitting function corresponding to the i-th view is determined as the color deviation fitting functions of other views in the GOP where the i-th view is located. The method according to any one of claims 16 to 20, characterized in that encoding the color cast fitting function comprises: The color cast fitting function is written into the patch data unit. The method according to any one of claims 16 to 20, characterized in that the i-th view is a non-basic view among the N views. A video decoding device, comprising: a determining unit, configured to decode a bitstream for an i-th view among N views, determine patch information of the i-th view, and generate a patch image of the i-th view based on the patch information, wherein the N views are views from N different viewpoints, N is a positive integer greater than 1, and i is a positive integer less than or equal to N; A dividing unit, used for dividing the patch image into M image blocks according to a preset block dividing method, where M is a positive integer; A decoding unit, configured to decode the bitstream for a j-th image block among the M image blocks to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer; A fitting unit is used to perform pixel fitting on the cropped pixels in the j-th image block using the P color deviation fitting functions to obtain a reconstructed block of the j-th image block. A video encoding device, comprising: a determining unit, configured to determine, for an i-th view among N views, a first pruning mask map of the i-th view, the first pruning mask map being a pruning mask map obtained by performing pixel pruning on the i-th view, the N views being views from N different viewpoints, N being a positive integer greater than 1, and i being a positive integer less than or equal to N; a dividing unit, configured to divide the i-th view into M image blocks based on a preset block dividing method, where M is a positive integer; a fitting unit, configured to determine, for a j-th image block among the M image blocks, unpruned pixels in the j-th image block based on the first pruning mask image, and perform chromaticity fitting on the unpruned pixels in the j-th image block to obtain P color deviation fitting functions of the j-th image block, where j is a positive integer less than or equal to M, and P is a positive integer; a pruning unit, configured to prune unpruned pixels in the j-th image block based on the P color cast fitting functions to obtain patch information of the i-th view; The encoding unit is used to encode the patch information and the P color deviation fitting functions to obtain a code stream. An electronic device, characterized in that it comprises a processor and a memory; The memory shown is used to store computer programs; The processor is used to call and run the computer program stored in the memory to implement any one of claims 1 to 15 or 16 to 37. The method described. A video encoding and decoding system, characterized in that it comprises: a video encoder and a video decoder; The video decoder is used to implement the method described in any one of claims 1 to 15 above; The video encoder is used to implement the method described in any one of claims 16 to 37. A computer-readable storage medium, characterized in that it is used to store a computer program; The computer program enables a computer to execute the method according to any one of claims 1 to 15 or 16 to 37 above.