RU2783348C1 - Encoder, decoder and corresponding methods for obtaining the boundary power of the debloking filter - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: invention relates to the field of image processing and, more specifically, to an encoder, a decoder and corresponding methods for obtaining the boundary power of a deblocking filter. A method for encoding and decoding is proposed. The method includes determining whether at least one of two blocks is predicted using combined inter-intra prediction (CIIP), in which the two blocks include a first block (Q block) and a second block (P block), and in which two blocks are associated with the border. Further includes setting the boundary power (Bs) of the boundary to a first value when at least one of the two blocks is predicted using CIIP; or setting the boundary power (Bs) of the boundary to a second value when neither of the two blocks is predicted using CIIP.

EFFECT: improving the efficiency of encoding/decoding.

45 cl, 17 dwg, 1 tbl

Description

The field of technology to which the invention belongs

Embodiments of the present invention (disclosure) mainly relate to the field of image processing and, more specifically, to an encoder, a decoder, and corresponding methods for obtaining a deblocking filter cutoff power.

State of the art

Video encoding (video encoding and decoding) is used in a wide range of digital video applications, such as digital television broadcast, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs. , video content collection and editing systems and security camera applications.

The amount of video data required to display even a relatively short video can be significant, which can make it difficult to transfer data over a communication network with limited bandwidth. Thus, in modern communication networks, video data is usually compressed prior to transmission. The size of the video also matters because memory resources may be limited to store the video on the storage device. Video compression devices often use source software and/or hardware to encode video data prior to transmission or storage, thereby reducing the amount of data required to represent digital video images. The compressed data is then received at the destination by a video decompressor, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, advanced compression and decompression technologies are required that increase the compression ratio with little or no degradation in image quality.

The essence of the invention

Embodiments of the present invention provide devices and methods for encoding and decoding in accordance with the independent claims.

The above and other objects of the present invention are solved by independent claims. Additional forms of implementation are obvious from the dependent claims, description and drawings.

According to a first aspect, the invention relates to an encoding method in which encoding includes decoding or encoding, the method includes: determining whether at least one of two blocks is predicted using CIIP (or MH) prediction, in which two blocks include a first block (Q block) and a second block (P block) and in which two blocks are associated with a boundary. The method also includes: setting a boundary power (Bs) of the boundary to a first value when at least one of the two blocks is a CIIP block; or setting the boundary power (Bs) of the boundary to a second value when neither of the two blocks is a CIIP block.

The method according to the first aspect of the invention can be performed by the device according to the second aspect of the invention. The device according to the second aspect of the invention includes a determination unit configured to determine whether at least one of the two blocks is predicted using combined inter-intra prediction (CIIP), in which the two blocks include the first block ( block Q) and a second block (block P) and in which two blocks are associated with the boundary. The device according to the second aspect of the invention also includes a setting unit configured to set the border power (Bs) of the border to a first value when at least one of the two blocks is predicted using CIIP, and set the border power (Bs) bounds to the second value when neither of the two blocks is predicted using CIIP.

Additional features and forms of implementation of the method according to the second aspect of the invention correspond to the features and forms of implementation of the device in accordance with the first aspect of the invention.

According to a third aspect, the invention relates to a device for decoding a video stream, which includes a processor and a memory. The memory stores instructions that cause the processor to execute the method according to the first aspect.

According to a fourth aspect, the invention relates to a device for encoding a video stream, which includes a processor and a memory. The memory stores instructions that cause the processor to execute the method according to the first aspect.

According to a fifth aspect, a computer-readable medium is provided that stores instructions thereon that, when executed, cause one or more processors to encode video data. The instructions cause one or more processors to execute a method in accordance with the first or second aspect, or any possible embodiment of the first aspect.

According to a sixth aspect, the invention relates to a computer program containing program code for executing a method on a computer according to the first aspect or any possible embodiment of the first aspect.

According to embodiments of the invention, by setting the edge power to a first value (e.g., set to 2) when at least one of the two blocks is a CIIP block, the deblocking filtering efficiency of block edges that are predicted using CIIP prediction is increased.

The accompanying drawings and description set forth a detailed description of one or more embodiments. Other features, objects and advantages will be apparent from the description, drawings and claims.

Brief description of the drawings

The following is a detailed description of embodiments of the present invention with reference to the accompanying drawings, in which:

fig. 1A is a block diagram showing an example of a video coding system capable of implementing embodiments of the invention;

fig. 1B is a block diagram showing another example of a video coding system capable of implementing embodiments of the invention;

fig. 2 is a block diagram showing an example of a video encoder capable of implementing embodiments of the invention;

fig. 3 is a block diagram showing an exemplary structure of a video encoder capable of implementing embodiments of the invention;

fig. 4 is a block diagram illustrating an example of an encoding device or a decoding device;

fig. 5 is a block diagram illustrating another example of an encoding device or a decoding device;

fig. 6A shows an example of all subblock boundaries in a CU using combined inter-intra prediction (CIIP);

fig. 6B shows an example of P and Q border blocks;

fig. 7 shows an example of a Bs production process in Embodiment 1;

fig. 8 shows an example of a process for obtaining Bs in VVC;

fig. 9 shows an example of a Bs production process in Embodiment 2;

fig. 10 shows an example of a Bs production process in Embodiment 3;

fig. 11 shows an example of deblocking all edges of a subblock in a CU that overlaps with (aligns with) an 8x8 sample grid;

fig. 12 shows an example of deblocking all edges of a subblock in a CU that overlaps with (aligns with) a 4x4 sample grid;

fig. 13 is a block diagram showing an example of a device structure for a power edge acquisition process;

fig. 14 is a block diagram showing an exemplary structure of a content delivery system 3100 that implements a content delivery service; and

fig. 15 is a block diagram showing an exemplary structure of a terminal device.

In the following description, identical reference numerals refer to identical or at least functionally equivalent features, unless expressly stated otherwise.

Description of Embodiments

In the following description, reference is made to the accompanying drawings, which form part of the present invention and show, by way of illustration, specific aspects of embodiments of the invention or specific aspects in which embodiments of the present invention may be used. Obviously, embodiments of the invention can be used in other aspects and contain structural or logical changes not shown in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

For example, it is obvious that the present invention with the described method can also be true for the corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more blocks, such as functional blocks, to perform the one or more method steps described (for example, one block performing one or more steps, or a plurality of blocks , each of which performs one or more of a plurality of steps) even if such one or more units are not described and illustrated explicitly in the drawings. On the other hand, for example, if a particular device is described in terms of one or more blocks, such as functional blocks, the corresponding method may include one step to perform the functionality of one or more blocks (for example, one step to perform the functionality of one or more blocks). blocks, or a plurality of steps, each of which performs the functionality of one or more of the plurality of blocks) even if such one or more steps is not explicitly described or illustrated in the drawings. In addition, it is understood that features of the various exemplary embodiments and/or aspects described herein may be combined with one another unless specifically noted otherwise.

Video coding generally refers to the processing of a sequence of images that form a video or video sequence. Instead of the term "image", the term "frame" or "image" can be used as synonyms in the field of video coding. The video coding used in the present invention (or the present disclosure) indicates either video coding or video decoding. Video encoding is performed at the source side, typically involving processing (eg, by compressing) the source video images to reduce the amount of data needed to represent the video images (for more efficient storage and/or transmission). Video decoding is performed on the destination side and usually contains inverse processing compared to the encoder for recovering video images. Embodiments relating to "encoding" video images (or images in general, as will be explained later) should be understood to refer to either "encoding" or "decoding" a video sequence. The combination of an encoding part and a decoding part is also called CODEC (Coding and Decoding).

In the case of lossless video coding, the original video images can be reconstructed, that is, the reconstructed video images have the same quality as the original video images (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, additional compression is performed, for example by quantization, to reduce the amount of data representing video images that cannot be fully reconstructed in the decoder, i.e., the quality of the reconstructed video images is lower or worse compared to the quality of the original video images.

Several video coding standards belong to the group of "hybrid lossy video codecs" (ie, combine spatial and temporal sample-domain prediction and 2D transform coding to apply transform-domain quantization). Each picture of a video sequence is usually divided into a set of non-overlapping blocks, and encoding is usually done at the block level. In other words, in the encoder, video is usually processed, that is, encoded, at the block (video block) level, for example, using spatial (inner picture) prediction and temporal (outer picture) prediction to generate a prediction block, subtracting the prediction block from the current block ( block currently being processed/to be processed) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder partially performs inverse processing compared to with the encoder to the encoded or compressed block to reconstruct the current block for presentation. In addition, the encoder duplicates the processing cycle of the decoder so that both will generate identical predictions (eg, intra and inter prediction) and/or re-restore for processing, ie, encoding, subsequent blocks.

In the following embodiments, the encoder 20, the decoder 30, and the encoding system 10 will be described with reference to FIGS. 1-3.

1A is a block diagram illustrating an example of a coding system 10, such as a video coding system 10 (or coding system 10 for short), that can use the methods of this invention. Video encoder 20 (eg, encoder 20) and video decoder 30 (eg, decoder 30) of video coding system 10 are examples of devices that can be configured to perform methods in accordance with various examples described in the present invention.

As shown in FIG. 1A, the encoding system 10 comprises a source device 12 configured to provide encoded image data 21 to, for example, destination device 14 for decoding encoded image data 13.

The source device 12 includes an encoder 20 and may additionally, i. e. possibly contain an image source 16, a preprocessor 18 (pre-processor), for example, an image preprocessor 18 and a communication interface or communication unit 22.

Image source 16 may comprise or be any type of image capture device, such as for capturing a real image and/or any kind of image or commentary (for screen encoding, some texts on the screen are also considered part of the image or image to be encoded), device generating, such as a computer graphics processor for generating a computer animated image, or any device for receiving and/or rendering a real image, a computer animated image (e.g., screen content, a virtual reality (VR) image), and/or any combination thereof (e.g., an image augmented reality (AR)). The image source can be any type of memory or storage that stores any of the above images.

Unlike the preprocessor 18 and the processing performed by the preprocessor 18, the image or image data 17 may also be referred to as raw image or original image data 17.

The preprocessor 18 is configured to receive (raw) image data 17 and perform pre-processing of the image data 17 to obtain a pre-processed image 19 or pre-processed image data 19. Pre-processing is performed by the preprocessor 18, which includes, for example, cropping, color format conversion (eg from RGB to YCbCr), color correction or denoising. Obviously, preprocessor 18 could be a possible component.

Video encoder 20 is configured to receive pre-processed image data 19 and provide encoded image data 21 (more details will be described below, for example, with reference to FIG. 2).

Communication interface 22 of source device 12 may be configured to receive encoded image data 21 and transmit encoded image data 21 (or any further processed version thereof) over communication channel 13 to another device, such as destination device 14 or any other device for storage or direct recovery.

Destination device 14 includes a decoder 30 (eg video decoder 30) and may additionally, i. e. possibly contain a communication interface or a communication unit 28, a post-processor 32 (post-processing unit 32) and a display device 34.

The communication interface 28 of the destination device 14 is configured to receive the encoded image data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. an encoded image data storage device, and provide encoded picture data 21 to decoder 30.

Communication interface 22 and communication interface 28 may be configured to transmit or receive encoded image data 21 or encoded data 13 via a direct link between source device 12 and destination device 14, such as a direct wired or wireless connection, or over any network, such as wired or wireless network or any combination thereof, or any private and public network or any combination thereof.

The communication interface 22 may, for example, be configured to pack the encoded image data 21 into an appropriate format, such as packets, and/or process the encoded image data using any kind of transmission coding or processing for transmission over a communication channel or communication network.

Communication interface 28 analogous to communication interface 22 may, for example, be configured to receive transmitted data and process transmitted data using any kind of appropriate transmission decoding or processing and/or decompression to obtain encoded image data 21.

Both the communication interface 22 and the communication interface 28 may be configured as unidirectional communication interfaces as shown by the arrow for the encoded image data 13 in FIG. 1A pointing from source device 12 to destination device 14 or as bidirectional communication interfaces, and may be configured, for example, to send and receive messages, for example, to establish a connection, acknowledge and exchange any other information related to the communication channel and/or data transmission, for example, the transmission of encoded image data.

The decoder 30 is configured to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (more details will be described below, for example, based on FIG. 3 or FIG. 5).

The post processor 32 of the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. Post-processing performed by block 32 post-processing may include, for example, color format conversion (for example, from YCbCr to RGB), color correction, cropping or resampling, or any other processing, for example, to prepare the decoded image data 31 for display, for example, using the display device 34.

The display device 34 of the destination device 14 is configured to receive the image data 33 after being processed to display the image to, for example, a user or viewer. The display device 34 may be or comprise any kind of display for presenting the reconstructed image, such as a built-in or external display or monitor. The displays may, for example, consist of liquid crystal displays (LCD), organic light emitting diode (OLED) displays, plasma displays, projectors, micro-LED displays, liquid crystals on silicon (LCoS), digital light processor (DLP), or any other display.

Although FIG. 1A depicts source device 12 and destination device 14 as separate devices, device embodiments may also comprise both or both source device 12 functionality or corresponding functionality and destination device 14 or corresponding functionality. In such embodiments, source device 12 or related functionality and destination device 14 or related functionality may be implemented using the same hardware and/or software, or using separate hardware and/or software, or any combination thereof.

As will be apparent to those skilled in the art based on the description, the presence and (exact) separation of functionality of various blocks or functions in source device 12 and/or destination device 14 as shown in FIG. 1A may vary depending on the actual device and implementation.

Encoder 20 (eg, video encoder 20) and decoder 30 (eg, video decoder 30), or both encoder 20 or decoder 30, may be implemented by a processing circuit as shown in FIG. 1B, such as one or more microprocessors, digital signal processors (DSPs). ), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. Encoder 20 may be implemented by processing circuitry 46 to implement various modules as discussed in relation to encoder 20 in FIG. 2 and/or any other coding system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to implement various modules as discussed with respect to decoder 30 in FIG. 3 and/or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations as described below. As shown in FIG. 5, if the methods are partially implemented in software, the device may store instructions for the software on a suitable non-volatile computer-readable storage medium and may execute the instructions in hardware using one or more processors to execute the methods of this invention. Either of video encoder 20 and video decoder 30 may be integrated as part of a combined encoder/decoder (CODEC) in one device, for example, as shown in FIG. 1B.

Source device 12 and destination device 14 may comprise any of a wide range of devices, including any kind of portable or stationary devices, such as laptops or laptops, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, game consoles, video streaming devices (eg, content service servers or content delivery servers), broadcast receiver, broadcast transmitter, and the like. and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

In some cases, the video coding system 10 illustrated in FIG. 1A is merely an example, and the methods of the present invention may be applied to video encoding settings (eg, video encoding or video decoding) that do not necessarily involve any communication between encoding and decoding devices. In other examples, data is retrieved from local memory, transferred over a network, and the like. The video encoding device may encode and store data in memory, and/or the video decoding device may retrieve and decode data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data in memory and/or retrieve and decode data from memory.

For convenience of description, embodiments of the invention are described herein, for example, with reference to High Efficiency Video Coding (HEVC) or Universal Video Coding (VVC) reference software, the next generation video coding standard developed by the Collaborative Video Coding Group (JCT-VC). ) ITU-T Video Coding Expert Group (VCEG) and ISO/IEC Moving Picture Expert Group (MPEG). For a person skilled in the art it is obvious that the embodiments of the invention are not limited to HEVC or VVC.

Encoder and encoding method

2 shows a block diagram of an exemplary video encoder 20 that is configured to implement the methods of the present invention. In the example of FIG. 2, video encoder 20 comprises an input 202 (or input interface 202), a residual calculation block 204, a transform processing block 206, a quantization block 208, an inverse quantization block 210, and an inverse transform processing block 212, a reconstruction block 214, a block 220 loop filter, decoded picture buffer (DPB) 230, prediction processing unit 260, entropy encoding unit 270, and output unit 272 (or output interface 272). The prediction processing block 260 may include an inter prediction block 244, an intra prediction block 254, and a mode selection block 262. Inter prediction block 244 may include a motion estimator and a motion compensation block (not shown). The video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

The residual calculator 204, the transform processing block 206, the quantization block 208, the prediction processing block 260 form the direct signal path of the encoder 20, while, for example, the inverse quantizer 210, the inverse transform processing block 212, the reconstruction block 214, the buffer 216, the loop filter 220, a decoded picture buffer (DPB) 230, inter predictor 244, and intra prediction 254 form an encoder signal return path, in which the encoder return signal path corresponds to the decoder signal path (see decoder 30 in FIG. 3). An inverse quantization block 210, an inverse transform processing block 212, a reconstruction block 214, a loop filter 220, a decoded picture buffer (DPB) 230, an inter prediction block 244, and an intra prediction block 254 are also related to the formation of the “embedded decoder” of the video encoder 20.

Images & image sharing (images and blocks)

Encoder 20 is configured to receive, for example, via input 201 an image 17 (or image data 17), for example an image of a sequence of images constituting a video or video sequence. The received image or image data may also be a pre-processed image 19 (or pre-processed image data 19). For ease of description, hereinafter referred to as picture 17. Picture 17 is also referred to as the current picture or the picture to be encoded (in particular, when encoding a video to distinguish the current picture from other pictures, for example, previously encoded and/or decoded pictures of one and the same video sequence, that is, a video sequence that also contains the current image).

A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. A sample in an array can also be called a pixel (short for picture element) or a pixel. The number of samples in the horizontal and vertical direction (or axis) of the array or image determines the size and/or resolution of the image. Three color components are typically used to represent a color, that is, an image can be represented or contain three arrays of samples. In RGB format or color space, an image contains a corresponding array of red, green, and blue color samples. However, in video encoding, each pixel is usually represented in a luma and chrominance format or color space, such as YCbCr, which contains a luma component labeled Y (sometimes L is also used instead) and two chrominance components labeled Cb and Cr. The Y component of luminance (or briefly, luminance) represents the brightness or intensity of the gray level (eg, as in a grayscale image), while the two chrominance components (or briefly, chrominance) Cb and Cr represent components of chrominance or color information. Accordingly, an image in YCbCr format contains an array of luma samples of luminance sample values (Y) and two chrominance sample arrays of chrominance values (Cb and Cr). RGB images can be converted to YCbCr format and vice versa, a process also known as color conversion. If the image is monochrome, it can only contain an array of brightness samples. Accordingly, the image may be, for example, an array of luminance samples in monochrome format, or an array of luminance samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color format.

Embodiments of encoder 20 may include a splitter (not shown in FIG. 2) configured to split image 17 into multiple (generally non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264/AVC), or coding tree blocks (CTBs) or coding tree blocks (CTUs) (H.265/HEVC and VVC). The partition block may be configured to use the same block size for all images of the video sequence and the corresponding grid defining the block size, or to change the block size between images, subsets or groups of images and divide each image into appropriate blocks.

In additional embodiments, the video encoder may be configured to directly receive block 203 of image 17, such as one, several or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

Like image 17, image block 203 is again, or can be viewed as, a two-dimensional array or matrix of samples with intensity values (sample values), albeit smaller than image 17. In other words, block 203 may contain, for example, one array of samples (for example , a luma array in the case of a monochrome image 17 or a luminance or chrominance array in the case of a color image) or three arrays of samples (for example, a luminance array and two chrominance arrays in the case of a color image 17) or any other number and/or types of arrays, depending on the applied color format. The number of samples in the horizontal and vertical direction (or axis) of block 203 determines the size of block 203. Accordingly, the block may, for example, be an array of MxN samples (M-column per N-row) or an MxN array of transform coefficients.

Embodiments of video encoder 20 as shown in FIG. 2 may be further configured to encode image 17 block by block, such as splitting and encoding block 203.

Remainder Calculation

The residual calculator 204 is configured to calculate the residual block 205 (also referred to as the residual 205) based on the image block 203 and the prediction block 265 (more details about the prediction block 265 are provided later), for example, by subtracting the sample values of the prediction block 265 from the sample values. image block 203, sample by sample (pixel by pixel) to obtain a residual block 205 in the sample area.

transformation

Transform processing unit 206 is configured to apply transforms, such as discrete cosine transform (DCT) or discrete sine transform (DST), on sample values of residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as residual transform coefficients and represent the residual block 205 in the transform region.

Transform processing unit 206 may be configured to apply DCT/DST integer approximations such as the transforms specified for HEVC/H.265. Compared to the orthogonal DCT, such integer approximations are usually scaled by a certain factor. To preserve the norm of the residual block, which is processed by direct and inverse transformations, additional scaling factors are applied during the transformation. Scaling factors are usually chosen based on certain constraints, such as scaling factors that are a power of two for the shift operation, the bit width of the transform coefficients, the trade-off between accuracy and implementation cost, and so on. Specific scaling factors, for example, are indicated for the inverse transform, for example, by the inverse transform processor 212 (and the corresponding inverse transform, for example, by the inverse transform processor 312 in the decoder 30), and the corresponding scaling factors for the forward transform, for example, by the transform processor 206 in encoder 20, respectively.

Embodiments of video encoder 20 (respectively, transform processing unit 206) may be configured to output transform parameters, such as the type of transform or transform, such as directly or encoded or compressed by entropy encoding unit 270, such that, for example, video decoder 30 may receive and use transformation parameters for decoding.

Quantization

Quantizer 208 is configured to quantize transform coefficients 207 to obtain quantized transform coefficients 209, for example by applying scalar quantization or vector quantization. The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209.

The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207 . For example, an n-bit transform factor may be rounded down to an m-bit transform factor during quantization, where n is greater than m. The degree of quantization can be changed by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a quantization parameter (QP). The quantization parameter may, for example, be an index to a predetermined set of applicable quantization step sizes. For example, small quantization parameters may correspond to fine quantization (small quantization step sizes), and large quantization parameters may correspond to coarse quantization (large quantization step sizes), or vice versa. Quantization may include division by the quantization step size and corresponding or inverse dequantization, for example by inverse quantizer 210, may include multiplication by the quantization step size. Embodiments according to some standards, such as HEVC, may be configured to use a quantization parameter to determine the quantization step size. Typically, the quantization step size can be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Additional scaling factors may be introduced for quantization and dequantization to recover the residual block norm, which may be modified due to the scaling used in the fixed-point approximation of the equation for the quantization step size and quantization parameter. In one exemplary implementation, inverse transform scaling and dequantization may be combined. Alternatively, customized quantization tables can be used and signaled from the encoder to the decoder, for example, in a bitstream. Quantization is a lossy operation in which the loss increases with increasing quantization step sizes.

Embodiments of video encoder 20 (respectively, quantizer 208) may be configured to output quantization parameters (QPs), for example, directly or encoded by entropy coding unit 270, such that, for example, video decoder 30 may receive and apply quantization parameters for decoding.

Inverse quantization

Inverse quantizer 210 is configured to apply the inverse quantization of quantizer 208 to the quantized coefficients to obtain dequantized coefficients 211, for example, by applying the inverse quantization scheme applied by quantizer 208 based on or using the same quantization step size as block 208 quantization. The dequantized coefficients 211 may also be referred to as the dequantized residual coefficients 211 and correspond, although not usually identical to the transform coefficients due to quantization loss, to the transform coefficients 207.

Reverse transformation

The inverse transform processing unit 212 is configured to apply an inverse transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the region samples. The recovered residual block 213 may also be referred to as a transformation block 213 .

Recovery

Restoration block 214 (e.g., adder or adder 214) is configured to add transform block 213 (i.e., reconstructed residual block 213) to prediction block 265 to obtain reconstructed block 215 in the sample area, for example, by addition, sample by sample, the sample values of the reconstructed residual block 213 and the sample values of the prediction block 265 .

Filtration

The loop filter block 220 (or "loop filter" 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221 or, typically, to filter the reconstructed samples to obtain filtered samples. The edge filter block is designed, for example, to smooth out pixel transitions or otherwise improve video quality. The loop filter block 220 may comprise one or more loop filters, such as a deblocking filter, an adaptive shift (SAO) filter, or other filters, such as a two-sided filter or an adaptive loop filter (ALF), or sharpening or smoothing filters, or collaborative filters or any combination of them. Although the loop filter block 220 is shown as a loop filter in FIG. 2, in other configurations, the loop filter block 220 may be implemented as a post loop filter. The filtered block 221 may also be referred to as the filtered reconstructed block 221.

Embodiments of video encoder 20 (respectively, loop filter unit 220) may be configured to output loop filter parameters (such as adaptive sample offset information), for example, directly or encoded by entropy encoding unit 270, so that, for example, decoder 30 may receive and apply the same loop filter parameters or corresponding loop filters for decoding.

Decoded picture buffer

Decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use in encoding video data by video encoder 20. DPB 230 may be formed by any of a variety of memories, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer (DPB) 230 is configured to store the filtered block 221. The decoded picture buffer 230 can be further configured to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221 of the same current picture or different pictures, for example , previously reconstructed images, and may provide complete previously reconstructed, i.e., decoded, images (and corresponding reference blocks and samples) and/or partially reconstructed current image (and corresponding reference blocks and samples), for example, for inter prediction. The decoded picture buffer (DPB) 230 is also configured to store one or more unfiltered reconstructed blocks 215 or, in general, unfiltered reconstructed samples, such as if the reconstructed block 215 is not filtered by the loop filter block 220 or any other further processed version of the reconstructed blocks or samples.

Mode selection (separation and prediction)

The prediction processing unit 260 comprises a mode selection unit 262, an inter prediction unit 244, and an intra prediction unit 254, and is configured to receive or obtain original image data, such as the original block 203 (the current block 203 of the current image 17) and reconstructed image data, for example, filtered and/or unfiltered reconstructed samples or blocks of the same (current) picture and/or from one or more previously decoded pictures, eg decoded picture buffer 230 or other buffers (eg, linear buffer, not shown). The reconstructed image data is used as reference image data for prediction, such as inter prediction or intra prediction, to obtain prediction block 265 or predictor 265.

Prediction processing unit 260 may be configured to determine or select a split for the current block prediction mode (including no split) and prediction mode (e.g., intra or inter prediction mode) and generate a corresponding prediction block 265 that is used to calculate the residual block. 205 and to restore the restored block 215.

Embodiments of prediction processor 260 may be configured to select a division and prediction mode (eg, those supported or available to prediction processor 260) that provide the best match, or in other words, minimum residual (minimum residual means better compression). for transmission or storage), or minimal signaling overhead (minimum signaling overhead means better compression for transmission or storage), or which takes into account or balances both. The prediction processing unit 260 may be configured to determine the division and prediction mode based on the rate distortion optimization (RDO), that is, select the prediction mode that provides the minimum rate distortion. Terms such as "best", "minimum", "optimum", etc. in this context does not necessarily refer to an overall "best", "minimum", "optimal", etc., but may also refer to termination performance or selection criteria such as a value above or below a threshold or other constraints , potentially leading to a "non-optimal choice" but reducing complexity and processing time.

In other words, the partitioner may be configured to partition the block 203 into smaller blocks or subblocks (which again form blocks), for example, iteratively using quadtree partitioning (QT), binary partitioning (BT), or ternary tree (TT), or any combinations thereof, and to perform, for example, prediction for each of the partitions of blocks or subblocks, wherein the mode selection comprises selecting a tree structure of the partitioned block 203 and the prediction modes are applied to each of the partitions or subblocks.

The following explains in more detail the splitting process (for example, splitter 260) and prediction processing (inter predictor 244 and intra predictor 254) performed by exemplary video encoder 20.

Separation

The partition block may divide (or split) the current block 203 into smaller sections, such as smaller square or rectangular blocks. These smaller blocks (which may also be referred to as sub-blocks) can be further subdivided into even smaller sections. This is also called tree splitting or hierarchical tree splitting, in which a root box, for example, at root level 0 of the tree (hierarchy level 0, depth 0) can be recursively split, such as split into two or more boxes of the next lower level of the tree, for example , nodes at level 1 of the tree (hierarchy level 1, depth 1) in which these blocks can be split again into two or more next lower level blocks, e.g. tree level 2 (hierarchy level 2, depth 2), etc. . until the split is complete, eg when a termination criterion is met, such as the maximum tree depth or minimum block size is reached. Blocks that are not split further are also called leaf blocks or leaf nodes of the tree. A tree using two partitions is called a binary tree (BT), a tree using three partitions is called a ternary tree (TT), and a tree using four partitions is called a quadtree (QT).

As mentioned earlier, the term "block" as used herein may be part, specifically, part of a square or rectangular shape of an image. With reference to, for example, HEVC and VVC, a block may either correspond to a coding tree block (CTU), a coding unit (CU), a prediction block (PU), and a transform unit (TU), and/or correspond to blocks, for example, a coding tree block (CTB), coding block (CB), transform block (TB), or prediction block (PB).

For example, a coding tree unit (CTU) may be either a luminance sample CTB, two corresponding chrominance sample CTBs of an image that has three sample arrays, or a monochrome image sample CTB or an image that is encoded using three separate color planes and the syntax structures used to encode samples. Accordingly, a coding tree block (CTB) may be a block of NxN samples for some value of N, such that splitting a component into CTBs is a split. A coding unit (CU) may be or comprise a luminance sample coding unit, two corresponding chrominance sample coding units of an image that has three sample arrays, or a monochrome image sample coding unit or an image that is encoded using three separate color planes and the syntax structures used to encode samples. Accordingly, a coding block (CB) may be a block of MxN samples for some values of M and N, so that the division of the CTB into coding blocks is a division.

In embodiments, such as in accordance with HEVC, a coding tree unit (CTU) may be partitioned into CUs using a square tree structure referred to as a coding tree. The decision as to whether to encode an image area using extrinsic (temporal) or intrinsic (spatial) prediction is made at the CU layer. Each CU can be further divided into one, two or four PUs according to the type of partition of the PU. Within one PU, the same prediction process is applied and the corresponding information is transmitted to the decoder based on the PU. After obtaining the residual block by applying a prediction process based on the division type of the PU, the CU may be divided into transform units (TUs) according to a different quadtree structure similar to the coding tree for the CU.

In embodiments, for example, in accordance with the latest video coding standard currently being developed, called universal video coding (VVC), square tree and binary tree (QTBT) splitting using coding block splitting. In the QTBT block structure, the CU can be either square or rectangular. For example, a coding tree unit (CTU) is first divided by a quadtree structure. The leaf nodes of a quadtree can be further subdivided using a binary tree or ternary (or ternary) tree structure. The leaf nodes of the quadtree are called coding units (CUs) and segmentation is used for prediction and transformation without any additional partitioning. This means that in most cases CU, PU and TU have the same block size in the QTBT coding block structure. In parallel, many partitions have also been proposed, for example, ternary tree partitioning has been used in conjunction with the QTBT block structure.

In one example, prediction processing unit 260 of video encoder 20 may be configured to perform any combination of the partitioning methods described herein.

As described above, encoder 20 is configured to determine or select the best or optimal prediction mode from a set of (predefined) prediction modes. The set of prediction modes may include, for example, intra prediction modes and/or inter prediction modes.

Internal prediction

The intra prediction mode set may contain 35 different intra prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes such as defined in HEVC, or may contain 67 different intra prediction modes such as , non-directional modes such as DC (or medium) mode and planar mode, or directional modes such as those defined for VVC.

The intra prediction block 254 is configured to use the reconstructed samples of adjacent blocks of the same current picture to generate the intra prediction block 265 according to the intra prediction mode of the intra prediction mode set.

The intra prediction block 254 (or, more generally, the prediction processing block 260) is further configured to output the intra prediction parameters (or, more generally, information indicative of the selected intra prediction mode for the block) to the entropy encoding block 270 in the form of syntax elements 266 for including an encoded image in data 21 so that, for example, video decoder 30 can receive and use prediction parameters for decoding.

external prediction

The set (or possible) inter prediction modes depends on the available reference pictures (i.e., previous at least partially decoded pictures, such as those stored in DBP 230) and other inter prediction parameters, such as whether the reference picture is all or only part of it. , for example, an area search box around the region of the current block, the reference frame is used to find the best match of the reference block and/or, for example, whether pixel interpolation is applied, such as half/half pixel, quarter pixel and/or 1/16 pixel interpolation or not.

In addition to the aforementioned prediction modes, a skip mode and/or a direct mode may be applied.

Inter prediction block 244 may include a motion estimator (ME) and a motion compensation (MC) block (both not shown in FIG. 2). The motion estimator is configured to receive or obtain an image block 203 (the current image block 203 of the current image 201) and a decoded image 231 or at least one or a plurality of previously reconstructed blocks, e.g., reconstructed blocks of one or a plurality of other/different previously decoded 231 images for motion estimation. For example. the video sequence may comprise the current picture and previously decoded pictures 231 or, in other words, the current picture and previously decoded pictures 231 may be part of or form a sequence of pictures that make up the video sequence.

The encoder 20 may, for example, be configured to select a reference block from a plurality of reference blocks of the same or different pictures from a plurality of other pictures and provide a reference picture (or reference picture index) and/or a shift (spatial shift) between position (x, y coordinates ) of the reference block and the position of the current block as inter-prediction parameters to the motion estimation block. This shift is also called motion vector (MV).

The motion compensation block is configured to receive, for example, receive an inter prediction parameter and perform inter prediction based on or using the inter prediction parameter to obtain inter prediction block 265 . The motion compensation performed by the motion compensator may comprise sampling or generating a prediction block based on the motion vector/block determined by the motion estimate, possibly performing interpolation to sub-pixel precision. Interpolation filtering can generate additional pixel samples from known pixel samples, thus potentially increasing the number of prediction block candidates that can be used to encode an image block. After receiving the motion vector for the PU of the current picture block, the motion compensator may locate the prediction block pointed to by the motion vector in one of the reference picture lists.

The motion compensation block may also generate syntax elements associated with blocks and a video segment for use by video decoder 30 when decoding image blocks of a video segment.

Entropy coding

The entropy encoding unit 270 is configured to apply an entropy encoding algorithm or schemes (e.g., variable length coding (VLC) schemes, context adaptive VLC scheme (CALVC), arithmetic coding scheme, context adaptive binary arithmetic coding (CABAC) based on context adaptive binary arithmetic coding (SBAC), probability interval entropy coding (PIPE), or other entropy coding technologies or methods) or bypass scheme (no compression) on quantized residual coefficients 209, inter prediction parameters, intra prediction parameters, and/or loop filter parameters and/or other syntax elements for obtaining encoded image data 21 that can be output by output 272, for example in the form of encoded bitstream 21, so that, for example, decoder 30 can receive and use parameters for decoding. The encoded bitstream 21 may be transmitted to video decoder 30 or stored in memory for later transmission or retrieval by video decoder 30.

Other structural variations of video encoder 20 may be used to encode the video stream. For example, non-transform-based encoder 20 may quantize the residual signal directly without transform processor 206 for specific blocks or frames. In another implementation, the encoder 20 may have a quantizer 208 and an inverse quantizer 210 combined into a single block.

Decoder and decoding method

Figure 3 shows an example of a video decoder 30, which is configured to implement the methods of the present invention. Video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), such as encoded by encoder 20, to obtain a decoded picture 331. The encoded picture data or bitstream contains information for decoding encoded picture data, such as data that are image blocks of the encoded video segment and associated syntax elements.

In the example of FIG. 3, the decoder 30 includes an entropy decoding block 304, an inverse quantization block 310, an inverse transform processing block 312, a reconstruction block 314 (e.g., an adder 314), a loop filter 320, a decoded picture buffer (DBP) 330, a block 344 external prediction and block 354 intra prediction. The inter prediction block may be or include a motion compensation block. Video decoder 30 may, in some examples, perform a decoding process that is generally the reverse of the encoding step described with respect to video encoder 100 in FIG.

As explained with respect to the encoder 20, the inverse quantization block 210, the inverse transform processing block 212, the reconstruction block 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction block 344, and the intra prediction block 354 are also called "inline decoder" generation. of the video encoder 20. Accordingly, the inverse quantizer 310 may be identical in function to the inverse quantizer 110, the inverse transform processor 312 may be identical in function to the inverse transform processor 212, the restorer 314 may be identical in function to the restorer 214, the loop filter 320 may be identical in function to the loop filter 220 and the buffer 330 of the decoded image may be identical in function to the buffer 230 of the decoded image. Therefore, the explanations provided for the respective blocks and functions of the video encoder 20 apply to the respective blocks and functions of the video decoder 30, respectively.

Entropy decoding

The entropy decoding unit 304 is configured to parse the bitstream 21 (or, more generally, the encoded image data 21) and perform, for example, entropy decoding of the encoded image data 21 to obtain, for example, quantized coefficients 309 and/or decoded coding parameters ( not shown in FIG. 3), such as any or all inter prediction parameters (eg, reference picture index and motion vector), intra prediction parameters (eg, intra prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and /or other syntax elements. Entropy decoding unit 304 may be configured to apply decoding algorithms or schemes corresponding to coding schemes as described with respect to entropy encoding unit 270 of encoder 20. Entropy decoding unit 304 may be further configured to provide inter prediction parameters, an intra prediction parameter, and/ or other syntax elements to mode select block 360 and other parameters to other blocks of decoder 30. Video decoder 30 may receive syntax elements at the video segment level and/or video block level.

Inverse quantization

The inverse quantizer 310 may be configured to receive quantization parameters (QPs) (or inverse quantization related information in general) and quantized coefficients from the encoded picture data 21 (e.g., by parsing and/or decoding, e.g., by entropy decoding block 304) and apply, based on the quantization parameters, inverse quantization to the decoded quantized coefficients 309 to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using a quantization parameter determined by video encoder 20 for each video block in a video segment to determine the degree of quantization, and likewise the degree of inverse quantization.

Reverse transformation

The inverse transform processor 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply transforms to the dequantized coefficients 311 to obtain reconstructed residual blocks 213 in the sample area. The restored residual blocks 213 may also be referred to as transform blocks. The transformation may be an inverse transformation, eg, inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transform processing unit 312 may be further configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., by parsing and/or decoding, e.g., by entropy decoding block 304) to determine the transform to be applied to the dequantized coefficients 311.

Recovery

A reconstructor 314 (e.g., adder 314) is configured to add the reconstructed residual block 313 to the prediction block 365 to obtain a reconstructed block 315 in the sample area, for example, by adding the sample values of the reconstructed residual block 313 and the sample values of the prediction block 365.

Filtration

The loop filter block 320 (either in the encoding loop or after the encoding loop) is configured to filter the reconstructed block 315 to obtain a filtered block 321, for example, to smooth pixel transitions or otherwise improve video quality. The loop filter block 320 may comprise one or more loop filters, such as a deblocking filter, an adaptive shift filter (SAO), or one or more other filters, such as a bidirectional filter, an adaptive loop filter (ALF), an anti-aliasing filter, a sharpening filter, or collaborative filters or any combination thereof. Although Figure 3 shows the loop filter block 320 as a loop filter, in other configurations, the loop filter block 320 may be implemented as a post loop filter.

Decoded image buffer

The decoded video blocks 321 in that frame or picture are then stored in the decoded picture buffer 330, which stores the decoded pictures 331 as reference pictures, used for subsequent motion compensation for other pictures and/or outputting the corresponding picture for display.

The decoder 30 is configured to output the decoded image 311, for example, through the output 312 for presentation or viewing to the user.

Prediction

Inter prediction block 344 may be identical to inter prediction block 244 (in particular, a motion compensation block) and intra prediction block 354 may be identical to inter function prediction block 254 and performs split or split decisions and predictions based on split and/or prediction parameters. or corresponding information received from the encoded image data 21 (eg, by parsing and/or decoding, eg, by entropy decoding block 304). The mode selector 360 may be configured to perform prediction (intra or inter prediction) for each block based on the reconstructed images, blocks, or corresponding samples (filtered or unfiltered) to obtain prediction block 365.

When a video segment is encoded as an intra-coded (I) segment, the intra prediction block 354 of the mode selector 360 is configured to generate a prediction block 365 for a picture block of the current video segment based on the signaled intra prediction mode and data from previously decoded blocks of the current picture. When a video image is encoded as an inter-coded (i.e., B or P) segment, the inter prediction block 344 (e.g., motion compensation block) of the mode select block 360 is configured to generate prediction blocks 365 for the video block of the current video segment based on motion vectors and others. syntax elements received from block 304 entropy decoding. For inter prediction, prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame lists, list 0 and list 1 using default construction methods based on reference pictures stored in DPB 330.

The mode selector 360 is configured to determine prediction information for the video block of the current video segment by analyzing motion vectors or other syntax elements, and uses the prediction information to generate prediction blocks for the current video block being decoded. For example, the mode selector 360 uses some of the received syntax elements to determine the prediction mode (eg, intra or inter prediction) used to encode the video blocks of a video segment, the inter prediction segment type (eg, B-segment, P-segment, or GPB- segment), construction information for one or more reference picture lists for the segment, motion vectors for each externally coded video block of the segment, inter prediction status for each externally coded video block of the segment, and other information for decoding video blocks in the current video segment.

Other variants of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter block 320 . For example, the non-transform-based decoder 30 can perform the inverse transform of the residual signal quantization directly without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, video decoder 30 may have an inverse quantizer 310 and an inverse transform processor 312 combined into one block.

It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after the interpolation filtering, motion vector output, or loop filtering, an additional operation such as clipping or shifting may be performed on the result of the interpolation filtering, motion vector output, or loop filtering processing.

It should be noted that additional operations can be applied to the received motion vectors of the current block (including, but not limited to, affine mode control point motion vectors, sub-block motion vectors of affine, planar, ATMVP modes, temporal motion vectors, etc.). . For example, the motion vector value is limited to a given range according to its representing bit. If the representing bit of the motion vector is bitDepth, then the range is -2^(bitDepth-1)~2^(bitDepth-1)-1, in which "^" means exponentiation. For example, if bitDepth is set to 16, then the range is -32768 ~ 32767; if bitDepth is set to 18, then the range is -131072 ~ 131071. For example, the value of the received motion vector (for example, MVs of four 4x4 subblocks within one 8x8 block) is limited such that the maximum difference between the integer parts of four 4x4 subblock MVs does not exceed N pixels, for example, no more than 1 pixel.

The present invention provides two methods for limiting a motion vector according to bitDepth.

Method 1: Remove MSB (Most Significant Bit) Overflow Through Expiration Operations

ux = (mvx + 2 ^bitDepth )% 2 ^bitDepth (1)

mvx = (ux >= 2 ^{bitDepth -1} )? (ux - 2 ^bitDepth ): ux(2)

uy = (mvy + 2 ^bitDepth )% 2 ^bitDepth (3)

mvy = (uy >= 2 ^{bitDepth -1} )? (uy - 2 ^bitDepth ): uy(4)

in which mvx is the horizontal component of the motion vector of the image block or sub-block, mvy is the vertical component of the motion vector of the image block or sub-block, and ux and uy indicate an intermediate value;

For example, if the mvx value is -32769, after formula (1) and (2) is applied, the resulting value is 32767. In a computer system, decimal numbers are stored as a binary component. The binary component of -32769 is 1.0111.1111.1111.1111 (17 bits) and the MSB is discarded, so the resulting binary component is 0111.1111.1111.1111 (decimal 32767), which is the same as the output value after applying formulas (1) and (2).

ux = (mvpx + mvdx + 2 ^bitDepth )% 2 ^bitDepth (5)

mvx = (ux >= 2 ^{bitDepth -1} )? (ux - 2 ^bitDepth ): ux(6)

uy = (mvpy + mvdy + 2 ^bitDepth )% 2 ^bitDepth (7)

mvy = (uy >= 2 ^{bitDepth -1} )? (uy - 2 ^bitDepth ): uy(8)

Operations can be applied during summation of mvp and mvp as shown in formula (5) to (8).

Method 2: Remove MSB overflow via value constraint

vx = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vx)

vy = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vy)

in which vx is the horizontal component of the motion vector of the image block or sub-block, vy is the vertical component of the motion vector of the image block or sub-block; x, y and z correspond to the three input values of the MV clipping process and the Clip3 function definition is as follows:

Clip3(x, y, z) =

4 is a diagram of a video encoding device 400 according to an embodiment of the present invention. The video encoding device 400 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, video encoding device 400 may be a decoder, such as video decoder 30 of FIG. 1A, or an encoder, such as video encoder 20 of FIG. 1A.

The video encoding device 400 includes input ports 410 and receiver (Rx) units 420 for receiving data; a processor, logical unit, or central processing unit (CPU) 430 for processing data; transmitter (Tx) blocks 440 and data output ports 450; and a memory 460 for storing data. The video encoder 400 may also include optical-electrical (OE) components and electrical-optical (EO) components connected to input ports 410, receiver units 420, transmitter units 440, and output ports 450 for outputting or inputting optical or electrical signals.

Processor 430 is implemented in hardware and software. Processor 430 may be implemented as one or more CPU chips, cores (eg, as a multi-core processor), FPGAs, ASICs, and DSPs. Processor 430 is connected to input ports 410, receiver units 420, transmitter units 440, output ports 450, and memory 460. Processor 430 includes an encoder 470. Encoding module 470 implements the disclosed embodiments described above. For example, encoding module 470 implements, processes, prepares, or provides various encoding operations. Thus, the use of the encoding module 470 provides a significant improvement in the functionality of the video encoding device 400 and causes the video encoding device 400 to be converted to another state. Alternatively, encoding module 470 is implemented as instructions stored in memory 460 and executed by processor 430.

Memory 460 includes one or more disks, tape drives, and solid state drives, and can be used as an overflow storage device, for storing programs when such programs are selected for execution, and for storing instructions and data that are read during program execution. Memory 460 may be volatile and/or non-volatile and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and/or static random access memory (SRAM).

FIG. 5 is a simplified block diagram of a device 500 that can be used as either or both source device 310 and destination device 320 of FIG. 1 according to an exemplary embodiment.

Processor 502 in device 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or a plurality of devices capable of manipulating or processing information that currently exists or will be developed in the future. Although the disclosed implementations may be implemented with a single processor, as shown with processor 502, for example, speed and efficiency advantages may be achieved using more than one processor.

In an implementation, memory 504 in device 500 may be read only memory (ROM) or random access memory (RAM). The memory 504 may be any other suitable type of storage device. Memory 504 may include code and data 506 accessed by processor 502 using bus 512. Memory 504 may further include an operating system 508 and application programs 510, application programs 510 include at least one program, which allows the processor 502 to perform the methods described herein. For example, application programs 510 may include applications 1 through N, which further include a video encoding application that performs the methods described herein. The device 500 may also include one or more output devices such as a display 518. The display 518 may be, in one example, a touch display that combines a display with a touch element capable of recognizing touch inputs. Display 518 may be connected to processor 502 via bus 512.

Although depicted as a single bus, bus 512 of device 500 may be comprised of multiple buses. In addition, secondary storage 514 may be directly connected to other components of the device 500 or may be accessed via a network and may contain one integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, device 500 can be implemented in a wide variety of configurations.

Combined Outer-Inner Prediction (CIIP)

By convention, a coding block is either intra-predicted (ie, using reference samples in the same picture) or externally predicted (ie, using reference samples in other pictures). Multi-hypothetical prediction combines these two prediction approaches. Hence, sometimes referred to as combined inter-intra prediction (CIIP). When using combined inter-intra prediction, weights are applied to the intra-predicted and inter-predicted samples and the final prediction is the weighted average of the samples.

A flag is used to indicate the use of combined inter-intra prediction (CIIP).

The block, using CIIP, can be further divided into several sub-blocks, as shown in figa. In one example, sub-blocks are obtained by dividing a block in the horizontal direction, with each sub-block having the same width as the original block but ¼ of the original block's height.

In one example, sub-blocks are obtained by splitting a block in the vertical direction, with each sub-block having the same height as the original block but ¼ of the original block's width.

The CIIP prediction process causes blocking artifacts because more residual signals are typically used when performing intra prediction. Blocking artifacts occur not only at the boundaries of the CIIP block, but also at the edges of the subblock within the CIIP block, such as the vertical edge A, B, C of the subblock in FIG. 6A. The horizontal edges of the subblock can be identified, respectively.

Although blocking artifacts can occur both at CIIP boundaries and at edges within CIIP blocks, the distortion caused by these two boundaries can be different and may require different edge powers.

In the rest of this description, the following terminology is used:

CIIP blocks: coding blocks that are predicted by applying multi-hypothetical prediction (CIIP).

Intra-coded blocks: coding blocks that are predicted by applying intra prediction, but not CIIP prediction.

Inter-coded blocks: coding blocks that are predicted using inter-prediction but not CIIP prediction.

Deblocking filter and edge power

Along with the successful principle of hybrid block video coding, video coding schemes such as HEVC and VVC are also being applied. Using this principle, the image is first divided into blocks, and then each block is predicted using intra or inter prediction modes. These blocks are encoded relative to adjacent blocks and approximate the original signal with some degree of similarity. Since the encoded blocks are only approximate to the original signal, the difference between the approximations can lead to discontinuities in the prediction and transformation of the block boundaries. These gaps are attenuated by the deblocking filter.

Bitstream information such as prediction modes and motion vectors is used to make a block boundary filtering decision. Some coding conditions are likely to cause strong blocking artifacts, which are represented by a so-called boundary power variable (Bs or BS) that is assigned to each block boundary and determined according to Table 1.

Table 1

Terms Bs At least one of the neighboring blocks is intra-coded 2 At least one of the neighboring blocks has non-zero transform coefficients one The absolute difference between motion vectors that belong to adjacent blocks is greater than or equal to one integer brightness sample one Motion prediction in neighboring blocks refers to different reference pictures or the number of motion vectors is different one Otherwise 0

Deblocking only applies to block boundaries with Bs greater than zero for the luma component and Bs greater than 1 for the chrominance components. Higher values of Bs provide stronger filtering using higher clipping parameter values. The conditions for obtaining Bs reflect the probability of the appearance of stronger blocking artifacts at the boundaries of internally predicted blocks.

Typically, the boundary of two adjacent blocks is denoted as P and Q, as shown in Fig.6B. Figv depicts the case of a vertical border. If a horizontal boundary is considered, then in FIG. 6B the image should be rotated 90 degrees clockwise, where P will be higher and Q lower.

Listing the Most Likely Mode

To improve coding efficiency in the intra prediction mode, a Most Likely Mode (MPM) list is used. Due to the large number of intra prediction modes (eg, 35 in HEVC and 67 in VVC), the intra prediction mode of the current block is not directly signaled. Instead, the most likely mode of the current block is listed based on its neighboring intra block prediction modes. As the intra prediction mode of the current block relative to its neighbors, the MPM list generally provides a high prediction level because its name is specified (Most Probable Mode list), thus the intra prediction mode of the current block is in the MPM list with high probability. Thus, only the list MPM index is signaled to obtain the intra prediction mode of the current block. Compared to the total number of intra prediction modes, the length of the MPM list is much smaller (for example, a 3-MPM list is used in HEVC and a 6-MPM list is used in VVC), so fewer bits are required for intra prediction mode coding. A flag (MPM_FLAG) is used to indicate whether the block's intra prediction mode belongs to its MPM list or not. If the assertion is true, then the intra prediction mode of the current block can be indexed using the MPM list. Otherwise, the intra prediction mode is directly signaled with a binarized code. In VVC and HEVC MPM, the list is based on its adjacent left and top blocks. When the left neighbor block and the top neighbor block of the current block are not available for prediction, the default mode list is used.

Motion vector prediction

Motion vector prediction is a technique used in encoding motion data. The motion vector typically has two components x and y, referring to movement in the horizontal and vertical directions, respectively. The motion vectors of the current block are usually correlated with the motion vectors of neighboring blocks in the current picture or in earlier coded pictures. Since neighboring blocks are quite likely to correspond to the same moving object with similar motion, and the motion of the object with low probability may change over time. Therefore, using the motion vectors in adjacent blocks as predictors reduces the size of the signaled motion vector difference. Motion vector predictors (MVPs) are typically derived from already decoded motion vectors from spatial neighbor blocks or from temporal neighbor blocks in a shared image.

When a block to be predicted is determined by applying CIIP prediction, its final predicted samples are based in part on the intra-predicted samples. Because, when performing intra prediction, residual and transformation coefficients are also used to a greater extent compared to blocks with inter prediction (mvd, merge, skip). Therefore, when these MH blocks are adjacent to other blocks, the boundary will be more heterogeneous. In HEVC and VVC, when either of two neighboring blocks of a boundary is intra-predicted, a strong deblocking filter is applied to that boundary, in which the boundary power parameter (Bs) is set to 2 (strongest).

In VTM3.0, however, the potential blocking artifact caused by predicted blocks via CIIP prediction is NOT considered. The power edge derivation process still uses blocks with CIIP predictions as externally predicted blocks. Under certain circumstances, this processing approach can lead to low subjective and objective quality.

In order to improve the deblocking filter, embodiments of the present invention offer several alternatives to using MH blocks, in which the process of deriving the edge power of a particular edge is influenced by MH blocks.

The normative document of the Universal Video Coding standard (draft 3) is defined as VVC draft 3.0 and can be found at the following link

http://phenix.it-sudparis.eu/jvet/doc_end_user/documents/12_Macao/wg11/JVET-L1001-v3.zip.

Embodiment 1

For a border with two sides (where the spatially adjacent blocks on each side are denoted as P block and Q block), the boundary power is defined as follows:

- As shown in FIG. 7, if at least one P and Q block is a CIIP prediction block, then the boundary power parameter of this boundary is set to a first value, for example, the first value may be 2.

- If both P and Q blocks are not predicted by applying CIIP prediction, and if at least one of the Q block or P block is predicted by applying intra prediction, the edge power is determined to be 2.

- If both P and Q blocks are not predicted by applying CIIP prediction, and if both Q and P blocks are predicted by applying inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates), the process of obtaining the boundary power of this boundary is shown in Fig.7.

- For comparison, the method specified in the VVC or ITU-H.265 video coding standard is shown in Fig.8.

- The pixel samples contained in the Q block and the P block are filtered using a deblocking filter in accordance with a certain boundary power.

Embodiment 2

As shown in Fig. 9, for a border with two sides (where the spatially adjacent blocks on each side are denoted as P block and Q block), the boundary power is obtained as follows:

- If at least one P and Q block is an intra predicted block, then the power edge is set to 2

Otherwise, if at least one P and Q block is a CIIP prediction block, then the boundary power parameter of that boundary is set to a first value, such as 1 or 2.

- Otherwise, if at least one of the neighboring blocks P and Q has non-zero transform coefficients. Then the boundary power parameter of this boundary is set to the second value, for example, 1

- Otherwise, if the absolute difference between the motion vectors that belong to blocks P and Q is greater than or equal to one integer brightness sample, then the boundary power parameter of this boundary is set to the second value, for example, 1.

- Otherwise, if the motion prediction in adjacent blocks refers to different reference pictures or the number of motion vectors is different, then the boundary power parameter of this boundary is set to 1

- Otherwise, the power limit parameter of this limit is set to 0.

- The pixel samples contained in the Q block and the P block are filtered using a deblocking filter in accordance with a certain boundary power.

Embodiment 3

As shown in FIG. 10, for a border with two sides (where the spatially adjacent blocks on each side are denoted as P block and Q block), the boundary power parameter of this border is set as follows:

- If at least one P and Q block is predicted using intra prediction and not using CIIP prediction (possibilities include P block predicted by intra prediction rather than predicted by multi-hypothetical prediction, and Q block predicted by any prediction function and vice versa ), then the power limit is set to 2.

- If both Q and P blocks are predicted either using inter prediction or using CIIP prediction (possibilities include a P block which is an inter predicted block and a Q block is an inter predicted block, or alternatively a P block is an inter predicted block and the Q block is an MH block, or alternatively the P block is an MH block and the Q block is an externally predicted block, or alternatively, the P block is an MH block and the Q block is an MH block).

- If at least one of the blocks P and Q has non-zero transform coefficients, then the boundary power parameter of said boundary is set to 1.

- Otherwise (if the Q and P blocks do not have non-zero transform coefficients), if the absolute difference between the motion vectors that are used to predict the P and Q blocks is greater than or equal to one integer sample, then the boundary power parameter of this boundary is set to 1.

- Otherwise (if the Q and P blocks do not have non-zero transform coefficients, and the absolute difference between the motion vectors is less than 1 sample), if the P and Q blocks are predicted based on different reference pictures, or the number of motion vectors that are used to predict the Q block and block P are not equal, then the boundary power parameter of this boundary is set to 1.

- Otherwise (if the above 3 conditions are evaluated as false), then the boundary power parameter of this boundary is set to 0.

- If at least one P and Q block is a CIIP block

prediction, then the boundary power is modified as follows:

- If the limit power is not equal to the predetermined first value (in one example, the predetermined first value is 2), then the limit power is increased by the predetermined second value (in one example, the predetermined second value is 1).

- The pixel samples contained in the Q block and the P block are filtered with

applying a deblocking filter in accordance with a certain boundary power.

Embodiment 4

For a boundary with two sides (P and Q, as described in VVC Draft 3.0), the boundary power is obtained as follows:

- If this boundary is a horizontal boundary and P and Q

belong to different CTUs, then

- If the Q block is a CIIP prediction block, then the power limit is set to 2.

- Otherwise, power limit is obtained as defined in VVC Draft 3.0

- Otherwise,

- If at least one P and Q block is a CIIP prediction block, then the boundary power parameter of that boundary is set to 2.

Otherwise, the boundary power of this boundary is obtained according to VVC Draft 3.0.

Option 5 implementation

For a border with two sides (where the spatially adjacent blocks on each side are denoted as a P block and a Q block), the boundary power is defined as follows:

- If at least one of the blocks P block or Q block is predicted with

applying intra prediction and not applying CIIP prediction (possibilities include P block predicted by intra prediction and not predicted by multi-hypothetical prediction, and Q block predicted by any prediction function and vice versa), then the boundary power is set to 2.

- If both blocks are predicted using either inter prediction or CIIP prediction (features include a P block that is an inter predicted block and a Q block that is an inter predicted block, or alternatively a P block that is an inter predicted block and Q block is MH block, or alternatively P is an MH block and Q block is an externally predicted block, or alternatively P block is an MH block and Q block is an MH block),

- If said boundary is a horizontal boundary and P and Q are located in two different CTUs, then

- If a Q block (where a Q block is denoted as a block that is downstream of a P block) is predicted by applying CIIP prediction, then the boundary power parameter of said boundary is set to 1.

- Otherwise (if block Q is not predicted using CIIP prediction), if at least one of the neighboring blocks P and Q has non-zero transform coefficients, then the boundary power parameter of said boundary is set to 1

Otherwise, if the absolute difference between the motion vectors that are used to predict the P and Q blocks is greater than or equal to one integer luma sample, then the boundary power parameter of said boundary is set to 1.

Otherwise, if compensated motion prediction is performed in adjacent P and Q blocks based on different reference pictures, or if the number of motion vectors that are used to predict Q and P blocks is not equal, then the boundary power parameter of said boundary is set to 1

- Otherwise, the boundary power parameter of said boundary is set to 0.

- Otherwise (if said boundary is a vertical

boundary or, if the Q block and the P block are contained within the same CTU),

- If at least one of the P and Q blocks is predicted by applying CIIP prediction, then the boundary power parameter of said boundary is set to 1.

- Otherwise, if at least one of the adjacent blocks P and Q has non-zero transform coefficients, then the boundary power parameter of said boundary is set to 1

Otherwise, if the absolute difference between the motion vectors that are used to predict the P and Q blocks is greater than or equal to one integer luma sample, then the boundary power parameter of said boundary is set to 1.

Otherwise, if compensated motion prediction is performed in adjacent P and Q blocks based on different reference pictures, or if the number of motion vectors that are used to predict Q and P blocks is not equal, then the boundary power parameter of said boundary is set to 1

- Otherwise, the boundary power parameter of this boundary is set to 0.

- The pixel samples contained in the Q block and the P block are filtered when

applying a deblocking filter in accordance with a certain boundary power.

Embodiment 6

For a border with two sides (where the spatially adjacent blocks on each side are denoted as a P block and a Q block), the boundary power is defined as follows:

- First, determine the boundary power of the said boundary in accordance with

with the method specified in the VVC or ITU-H.265 video coding standard.

- If said boundary is a horizontal boundary and P and Q

located in two different CTUs, then

- If the Q block is predicted using CIIP prediction, the boundary power is modified as follows:

- If the power limit is not equal to 2, then the power limit is increased by 1.

- Otherwise (if said boundary is a vertical

boundary or, if the Q block and the P block are contained within the same CTU),

- If at least one of the P block or the Q block is predicted using CIIP prediction, then the boundary power parameter of said boundary is adjusted as follows:

- If the power limit is not equal to 2, then the power limit is increased by 1.

- The pixel samples contained in the Q block and the P block are filtered with

applying a deblocking filter in accordance with a certain boundary power.

Embodiment 7

For a border with two sides (where the spatially adjacent blocks on each side are denoted as a P block and a Q block), the boundary power is obtained as follows:

- If said boundary is a horizontal boundary and blocks P and

Q are located in different CTUs, then

- If a Q block (where a Q block is denoted as a block that is downstream of the P block) is predicted using CIIP prediction, then the power edge is set to 2.

- If the Q block is not predicted using CIIP prediction and if at least one of the Q block or the P block is predicted using intra prediction, the edge power is determined to be 2.

- If Q blocks are not predicted using CIIP prediction and if both Q and P blocks are predicted using inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates).

- Otherwise (if said boundary is a vertical

boundary, or if the Q block and the P block are contained within the same CTU),

- If at least one of the P or Q blocks is predicted using CIIP prediction, then the boundary power parameter of said boundary is set to 2.

- If both P and Q block are not predicted using CIIP prediction and if at least one of the Q block or P block is predicted using intra prediction, the edge power is determined to be 2.

- If both P and Q blocks are not predicted using CIIP prediction, and if both Q and P blocks are predicted using inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates).

- The pixel samples contained in the q block and the P block are filtered when applying the deblocking filter according to a certain boundary.

Embodiment 8

For a border with two sides (where the spatially adjacent blocks on each side are denoted as P block and Q block), the boundary power is defined as follows:

- If at least one P and Q block is predicted using the internal

prediction and without applying CIIP prediction (capabilities include P block predicted by intra prediction and non-multi-hypothetical prediction and Q block predicted by any prediction function, and vice versa), then the boundary power is set to 2.

- If both Q and P blocks are predicted either using inter prediction,

either using CIIP prediction (features include a P block which is an externally predicted block and a Q block which is an externally predicted block, or alternatively a P block is an externally predicted block and a Q block is an MH block, or alternatively a P block, being an MH block and a Q block being an externally predicted block, or alternatively a P block being an MH block and a Q block being an MH block)

- If at least one of the blocks P and Q has non-zero transform coefficients, then the boundary power parameter of said boundary is set to 1

- Otherwise (if the Q and P blocks do not have non-zero transform coefficients), if the absolute difference between the motion vectors that are used to predict the P and Q blocks is greater than or equal to one integer sample, then the boundary power parameter of this boundary is set to 1.

- Otherwise (if the Q and P blocks do not have non-zero transform coefficients, and the absolute difference between the motion vectors is less than 1 sample), if the P and Q blocks are predicted based on different reference pictures or the number of motion vectors that are used to predict the Q block and block P are not equal, then the boundary power parameter of this boundary is set equal to 1.

- Otherwise (if the above 3 conditions evaluate to false), the boundary power parameter of this boundary is set to 0.

- If said boundary is a horizontal boundary and P and Q

located in two different CTUs, then

- If the Q block is predicted using CIIP prediction, then the defined boundary power is modified as follows:

- If the power limit is not equal to 2, then the power limit is increased by 1.

- If said boundary is a vertical boundary or if

block Q and block P are contained within the same CTU,

- If at least one of the P block and the Q block is predicted using CIIP prediction, then the boundary power parameter of said boundary is adjusted as follows:

- If the power limit is not equal to 2, then the power limit is increased by 1.

- The pixel samples contained in the Q block and the P block are filtered with

applying a deblocking filter in accordance with a certain boundary power.

Option 9 implementation

In one example, the boundary power (Bs) of CIIP block boundaries is set to 2, but the boundary power of sub-block boundaries within a CIIP is set to 1. When sub-block boundaries are not aligned to the 8x8 sampling grid, then the boundary power of such edges is set to 0. In FIG. .11 shows an 8x8 grid.

In another example, the boundary power of an edge is defined as follows:

For a border with two sides (where the spatially adjacent blocks on each side are denoted as a P block and a Q block), the boundary power is obtained as follows:

- If said boundary is a horizontal boundary and blocks P and

Q are located in different CTUs, then

- If a Q block (where a Q block is denoted as a block that is downstream of the P block) is predicted using CIIP prediction, then the power edge is set to 2.

- If the Q block is not predicted using CIIP prediction, and if at least one of the Q block or P block is predicted using intra prediction, the edge power is determined to be 2.

- If the Q block is not predicted by applying CIIP prediction, and if both Q and P blocks are predicted by applying inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates).

- Otherwise (if P and Q corresponding to two sub-blocks within CIIP

block, that is, the target boundary is a sub-block boundary within the CIIP block)

- If the sub-block border is aligned with the 8x8 grid, set the border power to 1.

- Otherwise (the subblock border is not aligned with the 8x8 grid), set the border power to 0.

- Otherwise (if said boundary is a vertical

boundary, or if Q and P are contained within the same CTU and P and Q are not in the same CIIP)

- If at least one of the P or Q blocks is predicted using CIIP prediction, then the boundary power parameter of said boundary is set to 2.

- 2. If both P and Q block are not predicted using CIIP prediction and if at least one of the Q block or P block is predicted using intra prediction, the edge power is determined to be 2.

- 3. If both P and Q blocks are not predicted using CIIP prediction, and if both Q and P blocks are predicted using inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates).

- The pixel samples contained in the Q block and the P block are filtered when

applying a deblocking filter in accordance with a certain boundary power.

Embodiment 10

In one example, the boundary power (Bs) of CIIP block boundaries is set to 2, but the boundary power of sub-block boundaries within the CIIP block is set to 1. 12 shows a 4x4 grid.

In another example, the boundary power of an edge is set as follows:

For a border with two sides (where the spatially adjacent blocks on each side are denoted as a P block and a Q block), the boundary power is obtained as follows:

- If said boundary is a horizontal boundary and blocks P and

Q are located in different CTUs, then

- If a Q block (where a Q block is denoted as a block that is downstream of the P block) is predicted using CIIP prediction, then the power edge is set to 2.

- If the Q block is not predicted using CIIP prediction and if at least one of the Q block or the P block is predicted using intra prediction, the edge power is determined to be 2.

- If the Q block is not predicted using CIIP prediction, and if both Q and P blocks are predicted using inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates).

- Otherwise (if P and Q corresponding to two sub-blocks within CIIP

block, that is, the target boundary is a sub-block boundary within the CIIP block)

- If the sub-block boundary is aligned with a 4x4 grid, the boundary power is set to 1.

- Otherwise (the sub-block boundary is not aligned with the 4x4 grid) the boundary power is set to 0.

- Otherwise (if said boundary is a vertical

boundary, or if Q and P are contained within the same CTU, and P and Q are not in the same CIIP)

- If at least one of the P or Q blocks is predicted using CIIP prediction, the boundary power parameter of said boundary is set to 2.

- If both P block and Q block are not predicted using CIIP prediction, and if at least one of the Q block or P block is predicted using intra prediction, the edge power is determined to be 2.

- If both P and Q blocks are not predicted using CIIP prediction, and if both Q and P blocks are predicted using inter prediction, then the edge power is determined to be less than 2 (the exact value of the edge power is determined according to additional conditional estimates).

- The pixel samples contained in the Q block and the P block are filtered with

applying a deblocking filter in accordance with a certain boundary power.

This application further provides the following embodiments:

Option 1 implementation. An encoding method, wherein the encoding includes decoding or encoding, and the method comprises:

determining whether or not the current coding block (or coding block) is predicted using the combined inter-intra prediction;

when the current coding block is predicted using combined inter-intra prediction,

setting a boundary power (Bs) of the boundary of the current coding block to a first value;

and

setting a power boundary (Bs) of a sub-coding block boundary to a second value, the current coding block contains at least two sub-coding blocks, and the sub-coding block boundary is a boundary between at least two sub-coding blocks.

Option 2 implementation. Method 1, wherein the method further comprises:

performing deblocking when the value of Bs is greater than zero for the luma component; or

performing deblocking when the value of Bs exceeds 1 for the chroma components, in which the value of Bs is one of the first value or the second value.

Option 3 implementation. The method of embodiment 1 or 2, wherein when the current coding block (or block) is predicted using combined inter-intra prediction, the current coding block is considered to be an intra-predictive block when deblocking is performed.

FIG. 13 is a block diagram showing an exemplary device structure 1300 for a power edge generation process. Device 1300 is configured to perform the above methods and may include:

a determination unit 1302, configured to determine whether at least one of the two blocks is predicted using combined inter-intra prediction (CIIP), in which the two blocks include a first block (Q block) and a second block (P block). ), and in which two blocks are associated with a boundary;

a setting unit 1304, configured to set the boundary power (Bs) of the boundary to a first value when at least one of the two blocks is predicted using CIIP; and set the boundary power (Bs) of the boundary to the second value when neither of the two blocks is predicted using CIIP.

For example, the setter is configured to set the bounds Bs to a first value when the first block is predicted using CIIP or when the second block is predicted using CIIP.

The device 2500 may further include a parser (not shown in FIG. 13) configured to parse the bitstream to obtain a flag, in which the flag is used to indicate whether at least one of the two blocks is predicted with using CIIP.

The determiner 1302 may be configured to determine whether at least one of the two blocks is predicted using intra prediction. The setter 1304 is configured to set Bs boundaries to a first value when neither of the two blocks is predicted using intra prediction and when at least one of the two blocks is predicted using CIIP. For example, the first value could be 1 or 2.

A setter 1304 is configured to set Bs boundaries to a second value when neither of the two blocks is predicted using intra prediction and when neither of the two blocks is predicted using CIIP. For example, the second value may be 1 when at least one of the two blocks (P and Q) has non-zero transform coefficients.

Invention Effect

Deblocking filtering of blocks predicted using multi-hypothetical prediction is performed with a medium power deblocking filter (cutoff power equal to 1).

If the block is predicted using CIIP prediction, then a first prediction using inter prediction is obtained and a second prediction is obtained using intra prediction, which are subsequently combined. Since the final prediction includes the intra-prediction part, there are usually more blocking artifacts in this part. Therefore, it is possible that there may be blocking artifacts at the boundaries of a block that is predicted by CIIP prediction. To overcome this shortcoming, the edge power is set to 2 in accordance with embodiments of the invention, which increases the probability of successful deblocking filtering of block edges that are predicted using CIIP prediction.

Additionally, embodiments of the present invention reduce the required amount of linear memory as follows. Linear memory is defined as the memory required to store information corresponding to the top line of the CTU, and which is needed in processing the adjacent bottom line of the CTU. For example, in order to filter a horizontal boundary between two rows of CTUs, prediction information (intra prediction/inter prediction/multi-hypothetical prediction) of the upper CTU must be stored in a memory list. Since 3 states (intra prediction/inter prediction/multi-hypothetical prediction) are possible to describe the block prediction mode, the linear memory requirement can be defined as 2 bits per block.

According to the embodiments of the present invention, if a block (block P in the embodiments) belongs to the top row of the CTU, then the deblocking operation only requires determination information whether the block is predicted using inter prediction or intra prediction (therefore, only 2 states can be stored with one bit per block). The following is the rationale for this assertion:

If the border between the P block and the Q block is a horizontal border, and if the Q block and the P block belong to 2 different CTUs (the Q block is one lower with respect to the P block in all embodiments), then the determination information whether the block is predicted with application of CIIP prediction is not used in determining the boundary power parameter. Therefore, it does not need to be stored. With embodiments of the invention, in a hardware implementation, the P block prediction mode can be temporarily changed as inter prediction (when P block is predicted using CIIP prediction), and edge power determination can be performed according to the changed prediction mode. After that (after determining the boundary power), the prediction mode can be changed back to CIIP prediction. It should be noted that hardware implementations are not limited to this method described herein (changing P block prediction mode at CTU boundary), but are simply provided as an example to clarify that, according to embodiments of the invention, the information of determining whether P block using CIIP prediction is not required when determining power edge (at horizontal CTU boundaries).

Therefore, according to embodiments of the invention, the amount of linear memory required is reduced from 2 bits per block to 1 bit per block. It should be noted that the total amount of linear memory required for implementation in hardware is proportional to the image and inversely proportional to the minimum block width.

It is noted that in accordance with all the previously described embodiments of the present invention, if a block is predicted using CIIP prediction, a first prediction is obtained by applying inter prediction and a second prediction is obtained by applying intra prediction, which are subsequently combined.

The above embodiments indicate that CIIP blocks are considered as intra prediction blocks to varying degrees when the deblocking filter is executed. Embodiment 1, 2, and 3 are used in three different strategies to adjust the boundary power of the boundary. Embodiment 1 refers to MH blocks that are entirely intra prediction blocks. Therefore, in table 1, the condition of setting Bs to 2 is the same.

Embodiment 2 also considers a condition where the degree of distortion caused by MH blocks is not as high as intra prediction blocks. Therefore, the power boundary conditions are tested first for intra prediction blocks and then for CIIP blocks. However, when a CIIP block is detected, Bs is still 2.

Embodiment 3 refers to MH blocks partly as intra prediction blocks, where Bs is incremented by one if at least one neighboring boundary block is an MH block. If Bs is already 2, then the normal acquisition strategy is used and Bs is not changed.

8 illustrates a flowchart of the Bs acquisition process in VVC Draft 3.0. Fig.7, 9 and 10 depict changes in the process of obtaining Bs for options 1, 2 and 3 implementation, respectively.

It is worth noting that through options 1 and 2, the implementation reduces not only the potential distortion, but also reduces the processing logic. In Embodiments 1 and 2, if the P or Q block is an MH block, the coefficient and motion vector checks are no longer required, thereby reducing the delay for checking the state.

Options 4, 5, 6 implementation are the corresponding variations of options 1, 2 and 3 implementation, which considers a linear buffer memory. The main difference from embodiments 1, 2 and 3 is that when the two sides P and Q are located in different CTUs and the edge is horizontal, the MH check of the block is performed asymmetrically. Namely, the P side of the block (ie, the top side) is not checked, but only the Q side of the block (ie, the bottom side) is checked. Thus, no additional linear buffer memory is allocated to store the CIIP flag of the P side of the block that resides in the other CTU.

In addition to the above embodiments, one additional feature of the MH blocks may be that the MH blocks do not need to be considered as intra prediction blocks sequentially. In one example, when looking for motion vector predictors of a current block, if its neighboring blocks are an MH block, then the motion vector of those MH blocks can be considered as motion vector predictors. In this case, the inter-prediction information of the MH blocks is used, and therefore the MH blocks are no longer considered to be intra-predictive blocks. In another example, when compiling an MPM list for intra prediction blocks, adjacent MH blocks of the current block may be considered as not including intra prediction information. Therefore, when checking the availability of MH blocks for the MPM to list the current block, they are marked as NOT available. Note that the MH blocks mentioned in this paragraph are not limited to the MH blocks that are used to determine the Bs value of the deblocking filter.

In addition to the above six embodiments, one additional feature of MH blocks may be that MH blocks are considered to be intra-predictive blocks all the time. In one example, when searching for motion vector predictors of a current block, if its neighboring blocks are MH blocks, their motion vector is excluded from the motion vector predictors. In this case, the inter prediction information of the MH blocks is NOT used, and therefore the MH blocks are considered as intra prediction blocks. In another example, when compiling an MPM list for intra prediction blocks, adjacent MH blocks of the current block may be considered to include intra prediction information. Therefore, when checking the availability of MH blocks to compile the MPM list of the current block, they are marked as available. Note that the MH blocks mentioned in this paragraph are not limited to the MH blocks that are used to determine the Bs value of the deblocking filter.

The following is an explanation of the implementations of the encoding method and the decoding method as shown in the above embodiments, and the system using them.

14 is a block diagram showing a content supply system 3100 for implementing a content distribution service. This content delivery system 3100 includes a capture device 3102, a terminal device 3106, and optionally includes a display 3126. The capture device 3102 communicates with the terminal device 3106 over a communication line 3104. The communication link may include the communication channel 13 described above. Link 3104 includes, but is not limited to, WIFI, Ethernet, cable, wireless (3G/4G/5G), USB, or any combination thereof, and the like.

The capture device 3102 generates data and may encode the data in an encoding manner as shown in the above embodiments. Alternatively, the capture device 3102 may distribute the data to a streaming server (not shown in the drawings), and the server encodes the data and transmits the encoded data to the terminal device 3106. The capture device 3102 includes, but is not limited to, a camera, smartphone, or tablet, computer or laptop, videoconferencing system, PDA, vehicle-mounted device, or a combination thereof, etc. For example, capture device 3102 may include source device 12 as described above. When the data includes video, the video encoder 20 included in the capture device 3102 may actually perform video encoding processing. When the data includes audio (ie, voice), the audio encoder contained in the capture device 3102 may actually perform audio encoding processing. For some practical scenarios, the capture device 3102 distributes encoded video and audio data by multiplexing them together. For other practical scenarios, such as in a videoconferencing system, encoded audio data and encoded video data are not multiplexed. The capture device 3102 separately distributes the encoded audio data and the encoded video data to the terminal device 3106.

In the content delivery system 3100, the terminal device 310 receives and reproduces the encoded data. The terminal device 3106 may be a device capable of receiving and recovering data, such as a smartphone or tablet 3108, a computer or laptop 3110, a Network Video Recorder (NVR)/Digital Video Recorder (DVR) 3112, a television 3114, a set-top box (STB) 3118 videoconferencing system 3120 video surveillance system 3120 personal digital assistant (PDA) 3122 device 3124 installed on the vehicle, or a combination or the like, configured to decode the above encoded data. For example, terminal 3106 may include destination 14 as described above. When the encoded data includes video, priority is given to the video decoder 30 contained in the terminal to perform video decoding. When the encoded data includes audio, the audio decoder contained in the terminal is given priority to perform audio decoding processing.

For a display terminal such as a 3108 Smartphone or Tablet, 3110 Computer or Laptop, 3112 Network Video Recorder (NVR)/DVR (DVR), 3114 TV, 3122 Personal Digital Assistant (PDA), or 3124 Vehicle Mounted Device, the terminal may transmit the decoded data to its display. For a terminal device not equipped with a display, such as STB 3116, video conferencing system 3118 or video surveillance system 3120, it establishes contact with an external display 3126 to receive and display decoded data.

When each device in this system performs encoding or decoding, an image encoding device or an image decoding device may be used as shown in the above embodiments.

15 is a diagram showing the structure of an example terminal device 3106. After the terminal device 3106 receives a stream from the capturing device 3102, the protocol processing unit 3202 parses the transmission protocol of the stream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Real Time Streaming Protocol (HLS), MPEG-DASH, Real Time Transport Protocol (RTP), Messaging Protocol in real time (RTMP) or any combination thereof or the like.

After the protocol processing unit 3202 processes the stream, a stream file is generated. The file is output to the demultiplexer 3204 . A demultiplexer 3204 may separate the multiplexed data into encoded audio data and encoded video data. As described above, for some practical scenarios, such as in a videoconferencing system, encoded audio data and encoded video data are not multiplexed. In this situation, the encoded data is transmitted to the video decoder 3206 and audio decoder 3208 without using the demultiplexer 3204.

Through the demultiplexing processing, a video elementary stream (ES), audio ES, and possibly subtitles are generated. The video decoder 3206, which includes the video decoder 30 as explained in the above embodiments, decodes the ES video using the decoding method as shown in the above embodiments to generate a video frame, and supplies this data to the synchronization block 3212. An audio decoder 3208 decodes the audio ES to generate an audio frame and provides this data to a sync block 3212. Alternatively, the video frame may be stored in a buffer (not shown in FIG. 18) before being fed to the sync block 3212. Likewise, the audio frame may be stored in a buffer (not shown in FIG. 18) before being fed into the sync block 3212.

The sync block 3212 synchronizes the video frame and the audio frame and supplies video/audio to the video/audio display 3214. For example, the sync block 3212 synchronizes the presentation of video and audio information. The information may be encoded in a syntax using timestamps relating to the presentation of the encoded audio and visual data and timestamps relating to the delivery of the data stream itself.

If there is a subtitle in the stream, the subtitle decoder 3210 decodes the subtitles and synchronizes them with the video frame and audio frame and passes the video/audio/subtitles to the video/audio/subtitle display 3216.

The present invention is not limited to the above system, and in the above embodiments, either the image encoding device or the image decoding device may be included in another system such as an automobile system.

Although embodiments of the invention have been primarily described in terms of video coding, it should be noted that the embodiments of encoding system 10, encoder 20, and decoder 30 (and system 10, respectively) and other embodiments described herein can also be performed. capable of processing or encoding a still image, that is, processing or encoding a single image independently of any preceding or subsequent image, as in video encoding. In general, if the image encoding processing is limited to one image 17, only inter prediction blocks 244 (encoder) and 344 (decoder) may be unavailable. All other functions (also referred to as tools or technologies) of video encoder 20 and video decoder 30 can equally be used for still picture processing, e.g. /312, split 262/362, intra prediction 254/354, and/or loop filtering 220, 320 and entropy coding 270 and entropy decoding 304.

Embodiments of, for example, encoder 20 and decoder 30, as well as the functions described herein, for example, in relation to encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted over a communication medium as one or more instructions or code and executed by a hardware processor. The computer-readable medium may include a computer-readable medium that corresponds to a tangible medium, such as a data medium, or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another, for example, in accordance with a communication protocol. Thus, computer-readable media can typically correspond to (1) tangible computer-readable storage media that are non-volatile, or (2) communication media such as a signal or carrier wave. Storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the methods described in the present invention. The computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk or other magnetic storage devices, flash memory, or any other media which can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are considered as the communication medium. However, it should be understood that computer-readable storage media and storage media do not include connections, carrier waves, signals, or other temporary media, but instead refer to non-volatile, tangible storage media. A disc as used herein includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk, and a Blu-ray disc, where the discs usually reproduce data magnetically and the discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the aforementioned structure, or any other structure suitable for implementing the described methods. Additionally, in some aspects described herein, functionality may be provided within dedicated hardware and/or software modules capable of encoding and decoding or contained in a combined codec. Additionally, the methods may be fully implemented in one or more circuits or logic elements.

The methods of the present invention may be implemented in a wide variety of devices or appliances, including a wireless telephone, an integrated circuit (IC), or a set of ICs (eg, a chipset). The present invention describes various components, modules, or blocks to represent the functional aspects of devices capable of performing the disclosed technologies, but not necessarily requiring implementation in different hardware blocks. Rather, as described above, the various blocks may be combined into a codec hardware block, or provided by a set of cooperating hardware blocks, including one or more processors, as described above, in combination with suitable software and/or firmware.

Claims (75)

1. An encoding method, wherein the encoding includes decoding or encoding, and the method comprises: determining whether at least one of two blocks of an image is predicted using combined inter-intra prediction (CIIP), in which the two blocks include a first block (Q block) and a second block (P block), and in which two blocks are associated with border; and setting the boundary power (Bs) of the boundary to a first value when at least one of the two blocks is predicted using CIIP, in which the first value is 2; and setting the boundary power (Bs) of the boundary to the second value at least partially when neither of the two blocks is predicted using CIIP. 2. The method of claim 1, wherein setting the boundary Bs to a first value comprises: setting the boundary Bs to a first value when the first block is predicted using CIIP or when the second block is predicted using CIIP. 3. The method according to claim 1 or 2, in which the method further comprises: parsing the bitstream to obtain a flag, in which the flag is used to indicate whether at least one of the two blocks is predicted using CIIP. 4. A method according to any one of claims 1 to 3, wherein the method further comprises: determining whether at least one of the two blocks is predicted using intra prediction. 5. The method of claim 4, wherein the method comprises: setting the boundary Bs to a first value when neither of the two blocks is predicted using intra prediction and when at least one of the two blocks is predicted using CIIP. 6. The method of claim 4, wherein the method comprises: setting the boundary Bs to a second value when neither of the two blocks is predicted using intra prediction and when neither of the two blocks is predicted using CIIP. 7. A method according to any one of claims 1 to 6, wherein the second value is 1 when at least one of the two blocks (P and Q) has non-zero transform coefficients. 8. The method of claim 1, wherein the method further comprises: obtaining the initial Bs boundary based on the VVC project 3.0; in which, when the value of the original Bs is less than the threshold value, the first value is equal to the sum of the value of the original Bs and a constant; in which, when the value of the original Bs is not less than the threshold value, the first value is equal to the value of the original Bs. 9. The method of claim 8, wherein the threshold value is 2. 10. The method according to claim 8 or 9, wherein the constant is 1. 11. The method according to any one of claims 1 to 10, wherein the first value is different from the second value. 12. The method according to any one of claims 1-11, wherein Bs indicates the strength of the deblocking filter, the first block includes a luma component and a chrominance component, and the method further comprises: performing deblocking when the value of Bs is greater than zero for the luma component; or performing deblocking when the value of Bs exceeds 1 for the chroma components, in which the value of Bs is one of the first value or the second value. 13. The method according to any one of claims 1 to 12, wherein when the block is predicted using CIIP, the block is treated as an intra prediction block when deblocking is performed, and wherein the block is the first block or the second block. 14. The method according to any one of claims 1 to 13, wherein when the block is predicted using CIIP, the block is considered to be a block that is predicted using inter-prediction when listing the most likely mode (MPM), and in which the block is the first block or second block. 15. The method according to any one of claims 1 to 14, wherein when the block is predicted using CIIP, the block is considered as a block that is predicted using inter prediction when performing motion vector prediction (MVP), and wherein the block is the first block or second block. 16. The method according to any one of claims 1 to 15, wherein when a block is predicted using CIIP, the block is considered to be a block that is predicted using intra prediction in a Most Likely Mode List (MPM), and in which the block is the first block or second block. 17. The method according to any one of claims 1 to 16, wherein when the block is predicted using CIIP, the block is considered as a block that is predicted using intra prediction when performing motion vector prediction (MVP), and wherein the block is the first block or second block. 18. An encoding method in which encoding includes encoding or decoding, comprising: determining whether a picture boundary coding block is predicted using combined inter-intra prediction (CIIP); and setting the border power (Bs) of the border to a first value when the coding block is a block that is predicted using CIIP, in which the first value is 2; and setting the boundary power (Bs) of the boundary to the second value at least partially when the coding block is not a block that is predicted using CIIP. 19. The method of claim 18, wherein two blocks are associated with a boundary, wherein the two blocks include a first block (Q block) and a second block (P block), and the boundary coding block is the first block. 20. The method of claim 19, wherein when the boundary is a horizontal boundary, and wherein the two blocks belong to different coding tree blocks (CTUs), the first block belongs to the lower side of the boundary. 21. The method according to any one of claims 18-20, in which the method further comprises: parsing the bitstream to obtain a flag, in which the flag is used to indicate whether a boundary coding block is predicted using CIIP. 22. The method according to any one of claims 18-21, wherein the method further comprises: determining whether the boundary coding block is predicted using intra prediction. 23. The method of claim 22, wherein the method comprises: setting the boundary Bs to a first value when the coding block is not predicted using intra prediction and when the coding block is predicted using CIIP. 24. The method of claim 22, wherein the method comprises: setting the boundary Bs to a second value when the coding block is not predicted using intra prediction and is not predicted using CIIP in which the second value is 1 when the coding block has non-zero transform coefficients. 25. The method according to any one of claims 18-21, in which the method further comprises: obtaining the initial Bs boundary based on the VVC project 3.0; in which, when the value of the original Bs is less than the threshold value, the first value is equal to the sum of the value of the original Bs and a constant; in which, when the value of the original Bs is not less than the threshold value, in which the first value is equal to the value of the original Bs. 26. The method of claim 25, wherein the threshold value is 2. 27. The method according to claim 25 or 26, wherein the constant is 1. 28. The method according to any one of claims 18-27, wherein Bs indicates the strength of the deblocking filter, the first block includes a luma component and a chrominance component, and the method further comprises: performing deblocking when the value of Bs is greater than zero for the luma component; or performing deblocking when the Bs value exceeds 1 for the chrominance component. 29. The method according to any one of claims 18 to 28, wherein when a coding block is predicted using CIIP, the coding block is considered as a block that is predicted using intra prediction when deblocking is performed. 30. The method according to any one of claims 18 to 29, wherein when a coding block is predicted using CIIP, the coding block is considered to be a block that is predicted using inter-prediction in a Most Likely Mode List (MPM). 31. The method according to any one of claims 18 to 30, wherein when a coding block is predicted using CIIP, the coding block is considered as a block that is predicted using inter prediction when performing motion vector prediction. 32. The method according to any one of claims 18 to 31, wherein when a coding block is predicted using CIIP, the coding block is considered to be a block that is predicted using intra prediction in a Most Likely Mode List (MPM). 33. The method according to any one of claims 18 to 32, wherein when a coding block is predicted using CIIP, the coding block is considered as a block that is predicted using intra prediction when performing motion vector prediction. 34. The method according to any one of claims 18-33, wherein the first value is the same as or different from the second value. 35. A device for obtaining boundary power, in which the device is an encoder or decoder and the device contains: a determination unit configured to determine whether at least one of the two image blocks is predicted using combined inter-intra prediction (CIIP), in which the two blocks include a first block (Q block) and a second block (P block), and in which two blocks are associated with a border; a setting unit configured to set the boundary power (Bs) of the boundary to a first value when at least one of the two blocks is predicted using CIIP in which the first value is 2, and set the boundary power (Bs) of the boundary to a second value of at least at least partially when neither of the two blocks is predicted using CIIP. 36. The apparatus of claim 35, wherein the setter is configured to set the bounds Bs to a first value when the first block is predicted using CIIP or when the second block is predicted using CIIP. 37. The device according to claim 35 or 36, in which the device further comprises: a parser configured to parse the bitstream to obtain a flag, in which the flag is used to indicate whether at least one of the two blocks is predicted using CIIP. 38. An apparatus according to any one of claims 35 to 37, wherein the determining block is further configured to determine whether at least one of the two blocks is predicted using intra prediction. 39. The apparatus of claim 38, wherein the setter is configured to set Bs boundaries to a first value when neither of the two blocks is predicted using intra prediction and when at least one of the two blocks is predicted using CIIP. 40. The apparatus of claim 38, wherein the setter is configured to set Bs boundaries to a second value when neither of the two blocks is predicted using intra prediction and when neither of the two blocks is predicted using CIIP. 41. An apparatus according to any one of claims 35-40, wherein the second value is 1 when at least one of the two blocks (P and Q) has non-zero transform coefficients. 42. An encoder (20) containing a processing circuit for performing the method according to any one of claims 1-34. 43. Decoder (30) containing a processing circuit for performing the method according to any one of claims 1 to 34. 44. Decoder, containing: one or more processors; and a persistent computer-readable medium coupled to the processors and storing programs for execution by the processors, wherein the programs, when executed by the processors, configure the decoder to execute the method of any one of claims 1-34. 45. An encoder containing: one or more processors; and a persistent computer-readable medium coupled to the processors and storing programs for execution by the processors, wherein the programs, when executed by the processors, configure the encoder to execute the method of any one of claims 1-34.

Images