RU2817618C2 - Processing multiple image sizes and matching windows for resampling reference image when encoding video - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to image encoding techniques. Result is achieved by the fact that the decoding method includes receiving a first set of image parameters and a second set of image parameters, each of which relates to the same set of sequence parameters, wherein, when the first set of image parameters and the second set of image parameters have the same image width and image height, first set of image parameters and second set of image parameters have the same matching window values; and applying a match window to the current image corresponding to the first set of image parameters or the second set of image parameters.

EFFECT: high efficiency of encoding images.

21 cl, 14 dwg

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 62/871,493, filed July 8, 2019, Jianle Chen et al., entitled “Multiple Image Size and Correspondence Window Processing for Reference Image Resampling in Video Coding,” which is incorporated herein by reference in its entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0002] In general, the present invention describes methods for supporting multiple image sizes and conformance windows in video encoding. More specifically, the present invention ensures that image parameter sets that have the same image size also have the same matching window.

BACKGROUND OF THE ART

[0003] The amount of video data required to present even a relatively short video can be significant, which can lead to difficulties when the data must be streamed or otherwise transmitted over a communications network with limited bandwidth. Thus, video data is typically compressed before being transmitted over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, as memory resources may be limited. Video compression devices often use software and/or hardware at the source to encode video data before transmission or storage, thereby reducing the amount of data required to represent digital video images. The compressed data is then received by the recipient's video decompressor, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques are desired that improve compression rates with little or no loss in image quality.

SUMMARY OF THE INVENTION

[0004] The first aspect relates to a method for decoding an encoded video bitstream performed by a video decoder. The method includes receiving by a video decoder a first set of image parameters and a second set of image parameters, each of which refers to the same set of sequence parameters, wherein the first set of image parameters and the second set of image parameters have the same image width and image height values. , the first set of image parameters and the second set of image parameters have the same matching window values; and applying, by the video decoder, a matching window to the current image corresponding to the first set of image parameters or the second set of image parameters.

[0005] The method provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters that have the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0006] Possibly in any of the preceding aspects, another implementation of the aspect provides that the match window comprises a match window offset left, a match window right offset, a match window offset up, and a match window offset down.

[0007] Possibly in any of the preceding aspects, another implementation of the aspect provides decoding of the current image corresponding to the first set of image parameters or the second set of image parameters using inter-frame prediction after applying a matching window, wherein the inter-frame prediction is based on the resampled reference image.

[0008] Possibly in any of the preceding aspects, another embodiment of the aspect resamples a reference picture associated with a current picture corresponding to a first set of picture parameters or a second set of picture parameters using reference picture resampling (RPS).

[0009] Possibly in any of the preceding aspects, another implementation of the aspect provides that resampling the reference image changes the resolution of the reference image used for inter-frame prediction of the current image corresponding to the first set of image parameters or the second set of image parameters.

[0010] Possibly in any of the preceding aspects, another implementation of the aspect ensures that the image width and image height are measured in luminance samples.

[0011] Possibly in any of the preceding aspects, another implementation of the aspect provides for determining whether bi-directional optical flow (BDOF) is enabled for decoding an image based on the image width, image height, and the matching window between the current image and a reference image for the current Images.

[0012] Possibly in any of the preceding aspects, another implementation of the aspect provides for determining whether decoder-side motion vector refinement (DMVR) is enabled for decoding a picture based on the picture width, picture height, and match window of the current picture, and reference image for the current image.

[0013] Possibly in any of the preceding aspects, another embodiment of the aspect causes an image generated using the current block to be displayed on the display of an electronic device.

[0014] The second aspect relates to a method for encoding a video bitstream by a video encoder. The method includes creating, by a video encoder, a first set of image parameters and a second set of image parameters, each of which relates to the same set of sequence parameters, wherein the first set of image parameters and the second set of image parameters have the same image width and height values. images, the first set of image parameters and the second set of image parameters have the same matching window values; encoding, by a video encoder, the first set of image parameters and the second set of image parameters into a video bitstream; and storing, by the video encoder, the video bit stream for transmission to the video decoder.

[0015] The method provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters having the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0016] Possibly in any of the preceding aspects, another implementation of the aspect provides that the match window comprises a match window offset left, a match window right offset, a match window offset up, and a match window offset down.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect ensures that the image width and image height are measured in luminance samples.

[0018] Possibly in any of the preceding aspects, another embodiment of the aspect provides a video bitstream comprising a first set of image parameters and a second set of image parameters to a video decoder.

[0019] The third aspect relates to the decoding device. The decoding device includes a receiver configured to receive an encoded video bitstream; a memory connected to the receiver that stores instructions; and a processor coupled to the memory, wherein the processor is configured to execute instructions for causing the decoding device to: receive a first set of image parameters and a second set of image parameters, each of which relates to the same set of sequence parameters, wherein when the first set image parameters and the second set of image parameters have the same image width and image height, the first set of image parameters and the second set of image parameters have the same matching window values; and applying a match window to the current image corresponding to the first set of image parameters or the second set of image parameters.

[0020] The decoding apparatus provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters that have the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0021] Possibly in any of the preceding aspects, another implementation of the aspect provides a match window comprising a left offset match window, a right offset match window, an up offset match window, and a down offset match window.

[0022] Possibly in any of the preceding aspects, another implementation of the aspect provides decoding of the current image corresponding to the first set of image parameters or the second set of image parameters using inter-frame prediction after applying a matching window, wherein the inter-frame prediction is based on the resampled reference image.

[0023] Possibly in any of the preceding aspects, another implementation of the aspect provides a display configured to display an image created based on the current image.

[0024] The fourth aspect relates to the encoding device. The encoding device includes a memory containing instructions; a processor coupled to the memory, wherein the processor is configured to execute instructions to initiate an encoder to: generate a first set of image parameters and a second set of image parameters, each of which relates to the same set of sequence parameters, wherein when the first set image parameters and the second set of image parameters have the same image width and image height, the first set of image parameters and the second set of image parameters have the same matching window values; and encoding the first set of image parameters and the second set of image parameters into a video bitstream; and a transmitter coupled to the processor, wherein the transmitter is configured to transmit a video bitstream comprising a first set of image parameters and a second set of image parameters to the video decoder.

[0025] The encoder provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters having the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0026] Optionally, in any of the preceding aspects, another implementation of the aspect provides a match window comprising a left offset match window, a right offset match window, an up offset match window, and a down offset match window.

[0027] Perhaps in any of the preceding aspects, another embodiment of the aspect ensures that the width and height of the image are measured in luminance samples.

[0028] The fifth aspect relates to encoding equipment. The encoding equipment includes a receiver configured to receive an image for encoding or to receive a bit stream for decoding; a transmitter coupled to the receiver, wherein the transmitter is configured to transmit a bit stream to a decoder or to transmit a decoded image to a display; a memory connected to at least one of the receiver or transmitter, the memory being configured to store instructions; and a processor coupled to the memory, wherein the processor is configured to execute instructions stored in the memory to perform any of the methods disclosed herein.

[0029] Encoding hardware provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters having the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0030] Possibly in any of the preceding aspects, another embodiment of the aspect provides a display configured to display an image.

[0031] The sixth aspect relates to the system. The system includes an encoder; and a decoder coupled to the encoder, wherein the encoder or decoder includes a decoding apparatus, an encoding apparatus, or an encoding equipment disclosed herein.

[0032] The system provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters having the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0033] The seventh aspect relates to the encoding means. The encoding means includes receiving means configured to receive an image for encoding or to receive a bit stream for decoding; transmission means connected to the reception means, transmission means configured to transmit a bit stream to the decoding means or transmit a decoded image to the display means; a storage means connected to at least one of the receiving means or the transmitting means, the storage means configured to store instructions; and processing means coupled to the storage means, the processing means configured to execute instructions stored in the storage means to perform any of the methods disclosed herein.

[0034] The encoding engine provides methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters having the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0035] For clarity, any of the above embodiments may be combined with any one or more of the other above embodiments to create a new embodiment within the scope of the present invention.

[0036] These and other features will be better understood from the following detailed description when taken in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To provide a more complete understanding of the present invention, reference is made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0038] FIG. 1 is a flow diagram of an exemplary method for encoding a video signal.

[0039] FIG. 2 is a schematic diagram of an exemplary encoding and decoding system (codec) for video encoding.

[0040] FIG. 3 is a schematic diagram illustrating an example of a video encoder.

[0041] FIG. 4 is a schematic diagram illustrating an example of a video decoder.

[0042] FIG. 5 is a coded video sequence depicting the relationship between an intra random access point (IRAP) image with respect to leading and trailing images in decoding order and presentation order.

[0043] FIG. 6 illustrates an example of layered coding for spatial scalability.

[0044] FIG. 7 is a schematic diagram illustrating an example of unidirectional inter-frame prediction.

[0045] FIG. 8 is a schematic diagram illustrating an example of bidirectional inter-frame prediction.

[0046] FIG. 9 illustrates a video bitstream.

[0047] FIG. 10 illustrates the image division method.

[0048] FIG. 11 illustrates an embodiment of a method for decoding an encoded video bitstream.

[0049] FIG. 12 illustrates an embodiment of a method for encoding a coded video bitstream.

[0050] FIG. 13 is a schematic diagram of a video encoding device.

[0051] FIG. 14 is a schematic diagram of an embodiment of an encoding means.

DETAILED DESCRIPTION OF OPTIONS FOR IMPLEMENTING THE INVENTION

[0052] At the outset, it should be understood that while an illustrative implementation of one or more embodiments is presented below, the disclosed systems and/or methods may be implemented using any number of techniques currently known or existing. The invention is in no way intended to be limited to the exemplary implementations, drawings and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be varied within the scope of the appended claims along with the full scope of their equivalents.

[0053] The following terms are defined as follows unless otherwise used herein. In particular, the following definitions are intended to further clarify the present invention. However, the terms may be described differently in different contexts. Accordingly, the following definitions should be considered in addition to, and should not be construed as a limitation of, any other definitions of the descriptions provided for such terms herein.

[0054] A bitstream is a sequence of bits including video data that is compressed for transmission between an encoder and a decoder. An encoder is a device that is configured to use encoding processes to compress video data into a bit stream. A decoder is a device that is configured to use decoding processes to recover video data from a bitstream for display. An image is an array of luminance samples and/or an array of chrominance samples that create a frame or its field. The image that is being encoded or decoded may be called the current image for clarity of discussion.

[0055] A reference picture is a picture that contains reference samples that can be used when encoding other pictures by reference in accordance with inter-picture prediction and/or inter-layer prediction. The reference picture list is a list of reference pictures used for inter-picture prediction and/or inter-layer prediction. Some video coding systems use two reference picture lists, which may be referred to as the first reference picture list and the zero reference picture list. A reference picture list structure is an addressable syntax structure that contains a plurality of reference picture lists. Inter-picture prediction is a mechanism for encoding samples of a current picture by reference to said samples in a reference picture that is different from the current picture, where the reference picture and the current picture are at the same level. A reference picture list structure element is an addressable location in the reference picture list structure that indicates a reference picture associated with the reference picture list.

[0056] A slice header is a portion of the encoded slice containing data elements related to all video data in the mosaic represented in the slice. A picture parameter set (PPS) is a set of parameters that contains data related to the entire image. More specifically, a PPS is a syntax structure containing syntax elements that apply to zero or more entire encoded pictures, as defined by the syntax element found in each picture header. A sequence parameter set (SPS) is a set of parameters that contains data related to a sequence of images. An access unit (AU) is a set of one or more coded pictures associated with the same display time (for example, the same picture order counter) for output from the decoded picture buffer (DPB). ) (for example, for display to the user). A decoded video sequence is a sequence of images that have been reconstructed by the decoder in preparation for display to the user.

[0057] A match crop window (or simply a match window) refers to a window of image samples in the encoded video sequence output from the encoding process. The bitstream may provide matching window trimming parameters to indicate an output region of the encoded image. Image width is the width of the image measured in luminance samples. Image height is the height of the image measured in luminance samples. The match window offsets (eg, conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset) define the samples of images referencing the PPS that are output from the decoding process, in terms of a rectangular area specified by the output image coordinates.

[0058] Decoder-Side Motion Vector Refinement (DMVR) is a coding process, algorithm, or tool used to refine the motion vector or motion vectors for a predicted block. DMVR allows one to find a motion vector based on two motion vectors found for bidirectional prediction using a two-way template matching process. In the DMVR, a weighted combination of prediction coding blocks formed with each of the two motion vectors can be found, and the two motion vectors can be refined by replacing them with new motion vectors that best point to the combined prediction coding block. Bi-Direction Optical Flow (BDOF) is a coding process, algorithm, or tool used to refine motion or motion vectors for a predicted block. BDOF allows motion vectors for subcoding blocks to be found based on the gradient of the difference between two reference images.

[0059] Reference Picture Resampling (RPR) is the ability to change the spatial resolution of encoded images in the middle of a bitstream without the need for internal image encoding at the location of the resolution change. The resolution used here describes the number of pixels in a video file. That is, resolution is the width and height of the projected image, measured in pixels. For example, a video might have a resolution of 1280 (horizontal pixels) × 720 (vertical pixels). This is usually written simply as 1280×720 or shortened to 720px.

[0060] Decoder-side motion vector refinement (DMVR) is a process, algorithm, or encoding tool used to refine the motion vector or motion vectors for a predicted block. Bi-directional optical flow (BDOF), also known as bidirectional optical flow (BIO), is a coding process, algorithm, or tool used to refine motion or motion vectors for a predicted block. Reference picture resampling (RPR) is the ability to change the spatial resolution of encoded images in the middle of a bitstream without the need for internal image encoding at the point where the resolution is changed.

[0061] The following abbreviations are used in this document: Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Coded Video Sequence (CVS) , Joint Video Experts Team (JVET), Motion-Constrained Tile Set (MCTS), Maximum Transfer Unit (MTU), Network Abstraction Layer (NAL) ), picture order counter (POC, Picture Order Count), raw byte sequence payload (RBSP, Raw Byte Sequence Payload), sequence parameter set (SPS, Sequence Parameter Set), universal video coding (VVC, Versatile Video Coding) and worker draft (WD, Working Draft).

[0062] FIG. 1 is a flow diagram of an exemplary operating method 100 for encoding a video signal. In particular, the video signal is encoded in the encoder. The encoding process compresses the video signal using various mechanisms to reduce the size of the video file. The smaller file size allows the compressed video file to be transmitted to the user while reducing bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process usually mirrors the encoding process so that the decoder can reconstruct the video signal sequentially.

[0063] At step 101, the video signal is input to the encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, a video file may be captured by a video capture device, such as a video camera, and encoded to support real-time video streaming. A video file may include both an audio component and a video component. A video component contains a series of image frames that, when viewed in sequence, create the visual impression of movement. Frames contain pixels that are expressed in units of light, referred to here as luminance components (or luminance samples), and color, which are referred to as chrominance components (or color samples). In some examples, frames may also contain depth values to support 3D viewing.

[0064] At step 103, the video is divided into blocks. Dividing involves dividing the pixels in each frame into square and/or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be divided into Coding Tree Units (CTUs), which are blocks of a predetermined size (e.g. , sixty-four pixels by sixty-four pixels). CTU blocks contain luma and chrominance samples. Encoding trees can be used to partition a CTU into blocks and then recursively partition the blocks until configurations that support further encoding are reached. For example, the luminance components of a frame can be subdivided until the individual blocks contain relatively uniform illuminance values. Additionally, the chrominance components of a frame can be subdivided until the individual blocks contain relatively uniform color values. Accordingly, the division mechanisms vary depending on the content of the video frames.

[0065] At step 105, various compression mechanisms are used to compress the image blocks divided at step 103. For example, inter-frame prediction and/or intra prediction may be used. Inter-frame prediction is designed to take advantage of the fact that objects in a shared scene tend to appear in consecutive frames. Accordingly, a block representing an object in a reference frame does not need to be re-described in adjacent frames. In particular, an object such as a table may remain in a constant position for several frames. Therefore, the table is described once, and adjacent frames can refer to the reference frame. Matching mechanisms can be used to match objects across multiple frames. In addition, moving objects may be represented in multiple frames, for example due to object movement or camera movement. As a specific example, a video might show a car moving across the screen in multiple frames. Motion vectors can be used to describe such motion. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in the frame to the coordinates of the object in the reference frame. As such, inter-frame prediction can encode an image block in the current frame as a set of motion vectors indicating an offset from the corresponding block in the reference frame.

[0066] Intra prediction encodes blocks in a common frame. Intra prediction takes advantage of the fact that luma and chrominance components tend to cluster together in a frame. For example, a patch of green on a piece of wood tends to be located next to similar patches of green. Intra prediction uses several directional prediction modes (for example, thirty-three in HEVC), planar mode, and direct current mode. Directional modes indicate that the current block is similar/matches the samples of the neighboring block in the corresponding direction. Planar mode specifies that a sequence of blocks along a row/column (such as a plane) can be interpolated based on adjacent blocks at the edges of the row. In effect, planar mode refers to a smooth transition of light/color across a row/column by using a relatively constant slope as values change. The DC mode is used to smooth edges and indicates that a block is similar to/matches the average value associated with the samples of all neighboring blocks associated with the angular directions of the directional prediction modes. Accordingly, intra prediction blocks may represent image blocks as different relational prediction mode values instead of actual values. In addition, inter-frame prediction blocks may represent image blocks as motion vector values instead of actual values. In any case, in some cases the prediction blocks may not accurately represent the image blocks. Any differences are stored in residual blocks. Transformations can be applied to the remaining blocks to further compress the file.

[0067] At step 107, various filtering techniques may be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction discussed above may result in blocky images in the decoder. In addition, the block-based prediction scheme can encode a block and then recover the encoded block for later use as a reference block. The in-loop filtering scheme iteratively applies noise reduction filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to blocks/frames. These filters reduce such blocking artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference blocks, so that artifacts are less likely to create additional artifacts in subsequent blocks that are encoded based on the reconstructed reference blocks.

[0068] After the video signal is separated, compressed, and filtered, the resulting data is encoded into a bitstream at step 109. The bitstream includes the data described above as well as any signaling data necessary to support proper reconstruction of the video signal at the decoder. For example, such data may include section data, prediction data, residual blocks, and various flags providing encoding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. The bitstream may also be broadcast and/or multicast to multiple decoders. Creating a bitstream is an iterative process. Accordingly, steps 101, 103, 105, 107 and 109 may occur continuously and/or simultaneously across multiple frames and blocks. The order shown in FIG. 1 is presented for clarity and ease of discussion and is not intended to limit the video encoding process to a particular order.

[0069] The decoder receives the bitstream and begins the decoding process at step 111. Specifically, the decoder uses an entropy decoding circuit to convert the bitstream into appropriate syntax and video data. The decoder uses the syntax data from the bitstream to determine partitions for frames at step 111. The partition must correspond to the results of the block partition at step 103. The entropy encoding/decoding used at step 111 is now described. The encoder makes many choices in the compression process, such as the choice of schemes dividing into blocks of several possible options based on the spatial positioning of values in the input image(s). A large number of slots can be used to signal precise selection. In this context, a bean is a binary value that is treated as a variable (for example, a bit value that can vary depending on the context). Entropy coding allows the encoder to discard any parameters that are clearly not suitable for a particular case, leaving a set of valid options. Each valid option is then assigned a code word. The length of the codewords is based on the number of valid options (for example, one cell for two options, two cells for three or four options, etc.). The encoder then encodes the codeword for the selected option. This scheme reduces the size of codewords because codewords are as large as desired to unambiguously indicate a choice from a small subset of valid options, as opposed to unambiguously indicating a choice from a potentially large set of all possible options. The decoder then decodes the choice, determining the set of valid options in a similar manner to an encoder. Having determined the set of valid options, the decoder can read the codeword and determine the choice made by the encoder.

[0070] At step 113, the decoder performs block decoding. In particular, the decoder uses the inverse transform to generate residual blocks. The decoder then uses the residual blocks and corresponding prediction blocks to reconstruct the image blocks according to the division. The prediction blocks may include both intra prediction blocks and inter-frame prediction blocks generated in the encoder at step 105. The reconstructed image blocks are then allocated to frames of the reconstructed video signal in accordance with the partition data determined at step 111. The syntax for step 113 may also transmitted in the bitstream using entropy encoding as described above.

[0071] At step 115, frames of the reconstructed video signal are filtered, similar to step 107 in the encoder. For example, noise reduction filters, deblocking filters, adaptive edge filters, and SAO filters can be applied to frames to remove blocking artifacts. After frame filtering, the video signal may be displayed at step 117 for viewing by the end user.

[0072] FIG. 2 is a schematic diagram of an exemplary encoding and decoding (codec) system 200 for video encoding. In particular, the codec system 200 provides functionality to support an implementation of the method 100 of operation. Codec system 200 is generalized to depict components used in both an encoder and a decoder. The codec system 200 receives and splits the video signal as described in relation to steps 101 and 103 in the operating method 100, resulting in a split video signal 201. The codec system 200 then compresses the split video signal 201 into an encoded bitstream, acting as an encoder, as described in relation to steps 105, 107 and 109 in method 100. Acting as a decoder, codec system 200 generates a video output signal from the bitstream as described with respect to steps 111, 113, 115 and 117 in method 100 of operation. The codec system 200 includes a general encoder control component 211, a transform scaling and quantization component 213, an intra-image estimation component 215, an intra-image prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 227 filter control analysis, in-loop filter component 225, decoded image buffer component 223, and header formatting and context adaptive binary arithmetic coding (CABAC) component 231. Such components are connected as shown. In fig. The 2 black lines indicate the movement of data that must be encoded/decoded, and the dotted lines indicate the movement of control data that controls the operation of other components. All components of the codec system 200 may be present in the encoder. The decoder may include a subset of codec system 200 components. For example, the decoder may include an in-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are now described.

[0073] The divided video signal 201 is a captured video sequence that has been divided into blocks of pixels by a coding tree. The coding tree uses different partitioning modes to divide a block of pixels into smaller blocks of pixels. These blocks can then be divided into smaller blocks. Blocks may be called nodes in a coding tree. Larger parent nodes are split into smaller child nodes. The number of times a node is divided into subdivisions is called the node/encoding tree depth. In some cases, the divided blocks may be included in coding units (CUs). For example, a CU may be part of a CTU that contains a luminance block, chrominance red (Cr) block(s), and chrominance blue block(s) (Cb), along with corresponding syntax instructions for the CU. Partitioning modes can include binary tree (BT, triple tree), ternary tree (TT, triple tree) and quad tree (QT), used to split a node into two, three or four child nodes, respectively, of different shapes depending on the separation modes used. The divided video signal 201 is sent to a general encoder control component 211, a transform scaling and quantization component 213, an intra-image estimation component 215, a filter control analysis component 227, and a motion estimation component 221 for compression.

[0074] General encoder control component 211 is configured to make decisions related to encoding images of a video sequence into a bitstream in accordance with application constraints. For example, the encoder general control component 211 controls the optimization of bitrate/bitstream size versus reconstruction quality. Such decisions can be made based on disk space/bandwidth availability and image resolution requests. The general encoder control component 211 also controls buffer usage based on the bit rate to reduce buffer underrun and overflow problems. To manage these problems, the general encoder control component 211 controls separation, prediction, and filtering through other components. For example, the general encoder control component 211 may dynamically increase compression complexity to increase resolution and increase bandwidth utilization, or decrease compression complexity to decrease resolution and bandwidth utilization. Therefore, the general encoder control component 211 controls the other components of the codec system 200 to balance the quality of the video signal reconstruction with data rate issues. The encoder general control component 211 creates control data that controls the operation of other components. Control data is also sent to header formatting and CABAC component 231 for encoding into a bitstream to signal parameters for decoding at the decoder.

[0075] The divided video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. A frame or slice of divided video signal 201 may be divided into multiple video blocks. Motion estimation component 221 and motion compensation component 219 perform inter-frame prediction coding of a received video block with respect to one or more blocks in one or more reference frames to provide temporal prediction. Codec system 200 may perform multiple encoding passes, for example, to select an appropriate encoding mode for each block of video data.

[0076] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by motion estimation component 221 is a process for generating motion vectors that estimate motion for video blocks. The motion vector, for example, may indicate an offset of the encoded object relative to the prediction block. A prediction block is a block that is found to closely match the block to be encoded in terms of pixel difference. The prediction block may also be called a reference block. This pixel difference can be determined by the sum of absolute difference (SAD, sum of absolute difference), sum of squared difference (SSD, sum of square difference), or other difference measures. HEVC uses several encoded entities, including CTUs, coding tree blocks (CTBs), and CUs. For example, CTUs can be divided into CTBs, which can then be divided into CBs to be included in a CU. The CU may be encoded as a prediction unit (PU) containing the prediction data and/or a transform unit (TU) containing the transformed residual data for the CU. Motion estimation component 221 generates motion vectors, PU and TU, using speed distortion analysis as part of the speed distortion optimization process. For example, motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block/frame and can select reference blocks, motion vectors, etc. that have the best speed-distortion characteristics. The best rate-distortion performance balances both video restoration quality (eg, amount of data loss during compression) and encoding efficiency (eg, final encoding size).

[0077] In some examples, codec system 200 may calculate values for sub-integer pixel positions of reference pictures stored in decoded picture buffer component 223. For example, video codec system 200 may interpolate the values of quarter pixel positions, eighth pixel positions, or other fractional pixel positions of a reference image. Therefore, the motion estimation component 221 can search for motion with respect to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision. Motion estimation component 221 calculates a motion vector for a video block PU in the inter-frame coded slice by comparing the position of the PU with the position of the reference picture prediction block. The motion estimation component 221 outputs the calculated motion vector as motion data for the header formatting component 231 and CABAC for encoding and motion to the motion compensation component 219 .

[0078] The motion compensation performed by the motion compensation component 219 may include sampling or generating a prediction block based on the motion vector determined by the motion estimation component 221 . Again, in some examples, motion estimation component 221 and motion compensation component 219 may be functionally integrated. After receiving the motion vector for the PU of the current video block, motion compensation component 219 can locate the prediction block pointed to by the motion vector. A residual video block is then generated by subtracting the pixel values of the prediction block from the pixel values of the current video block being encoded, generating pixel difference values. In general, motion estimation component 221 performs motion estimation relative to luminance components, and motion compensation component 219 uses motion vectors calculated based on luminance components for both chrominance and luminance components. The prediction block and the residual block are sent to the transform scaling and quantization component 213 .

[0079] The divided video signal 201 is also sent to the intra-image estimation component 215 and the intra-image prediction component 217 . As with the motion estimation component 221 and the motion compensation component 219, the intra-image estimation component 215 and the intra-image prediction component 217 may be highly integrated but are illustrated separately for conceptual purposes. The intra-image estimation component 215 and the intra-image prediction component 217 predict the current block relative to blocks in the current frame as an alternative to the inter-frame prediction performed by the motion estimation component 221 and the inter-frame motion compensation component 219 as described above. In particular, the intra-image estimation component 215 determines the intra prediction mode to use for encoding the current block. In some examples, the in-image evaluation component 215 selects an appropriate intra-prediction mode for encoding the current block from a plurality of tested intra-prediction modes. The selected intra prediction modes are then sent to header formatting component 231 and CABAC for encoding.

[0080] For example, the intra-image evaluation component 215 calculates the speed distortion values using the speed distortion analysis for the various intra prediction modes tested, and selects the intra prediction mode having the best speed distortion characteristics among the tested modes. Rate distortion analysis typically determines the amount of distortion (or error) between the encoded block and the original uncoded block that was encoded to create the encoded block, as well as the data rate (e.g., number of bits) used to create the encoded block. Intra-image estimation component 215 calculates ratios of distortions and rates for various coded blocks to determine which intra prediction mode shows the best rate distortion value for the block. In addition, the intra-image estimation component 215 may be configured to encode the depth blocks of the depth map using a depth modeling mode (DMM) based on rate-distortion optimization (RDO).

[0081] The intra-image prediction component 217 may generate a residual block from the prediction block based on selected intra-prediction modes determined by the intra-image estimation component 215 when implemented on an encoder, or read the residual block from a bitstream when implemented on a decoder. The residual block includes the difference in values between the prediction block and the original block, represented as a matrix. The residual block is then sent to the transform scaling and quantization component 213 . The intra-image estimation component 215 and the intra-image prediction component 217 may operate on both luma and chrominance components.

[0082] The transform scaling and quantization component 213 is configured to further compress the residual block. Transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block, creating a video block containing the values of the residual transform coefficients. You can also use wavelet transforms, integer transforms, subrange transforms, or other types of transforms. The transform may transform residual information from a pixel domain into a transform domain such as a frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information so that different frequency information is quantized at different levels of detail, which can affect the final visual quality of the reconstructed video. Transform scaling and quantization component 213 is also configured to quantize transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be changed by adjusting the quantization parameter. In some examples, transform scaling and quantization component 213 may then scan a matrix including the quantized transform coefficients. The quantized transform coefficients are sent to header formatting component 231 and CABAC for encoding into the bitstream.

[0083] The scaling and inverse transform component 229 applies the inverse operation of the transform scaling and quantization component 213 to support motion estimation. The scaling and detransforming component 229 applies descaling, transforming, and/or quantization to reconstruct a residual block in a pixel domain, for example, for later use as a reference block that can become a prediction block for another current block. Motion estimation component 221 and/or motion compensation component 219 may calculate a reference block by adding a residual block back to the corresponding prediction block for use in motion estimation of a later block/frame. Filters are applied to the reconstructed reference blocks to reduce artifacts introduced during scaling, quantization, and transformation. Otherwise, such artifacts may cause inaccurate prediction (and create additional artifacts) when predicting subsequent blocks.

[0084] Filter control analysis component 227 and in-loop filter component 225 apply filters to residual blocks and/or reconstructed image blocks. For example, the transformed residual block from the upscaling and inverse transform component 229 may be combined with the corresponding prediction block from the intra-image prediction component 217 and/or the motion compensation component 219 to restore the original image block. Filters can then be applied to the reconstructed image block. In some examples, filters may be applied to residual blocks instead. As with the other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and may be implemented together, but are depicted separately for conceptual purposes. Filters applied to reconstructed reference blocks apply to specific spatial regions and include several options to customize how such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference blocks to determine where such filters should be applied and sets the appropriate parameters. Such data is sent to header formatting component 231 and CABAC as filter control data for encoding. The in-loop filter component 225 applies such filters based on filter control data. The filters may include a deblocking filter, a noise reduction filter, a SAO filter, and an adaptive loop filter. Such filters may be applied in the spatial/pixel domain (eg, in a reconstructed block of pixels) or in the frequency domain, depending on the example.

[0085] When operating as an encoder, the filtered reconstructed image block, residual block, and/or prediction block are stored in decoded image buffer component 223 for later use in motion estimation as described above. When operating as a decoder, the decoded image buffer component 223 stores and forwards the reconstructed and filtered blocks to the display as part of the output video signal. The decoded image buffer component 223 may be any memory device capable of storing prediction blocks, residual blocks, and/or reconstructed image blocks.

[0086] Header formatting and CABAC component 231 receives data from various components of codec system 200 and encodes such data into a coded bitstream for transmission to a decoder. Specifically, the header formatting and CABAC component 231 generates various headers for encoding control data such as general control data and filter control data. In addition, prediction data, including intra prediction and motion data, as well as residual data in the form of quantized transform coefficient data, are all encoded in the bitstream. The final bit stream includes all information required by the decoder to reconstruct the original divided video signal 201. Such information may also include intra prediction mode index tables (also called codeword mapping tables), definitions of encoding contexts for various blocks, indications of the most likely modes internal prediction, indicating partition information, etc. Such data can be encoded using entropy coding. For example, information can be encoded using context adaptive variable length coding (CAVLC), CABAC, based on the syntax of context-adaptive binary arithmetic coding (SBAC), entropy coding with probability interval partitioning entropy (PIPE) or another entropy coding method. After entropy encoding, the encoded bitstream can be transferred to another device (such as a video decoder) or archived for later transmission or retrieval.

[0087] FIG. 3 is a block diagram illustrating an example video encoder 300. Video encoder 300 may be used to implement encoding functions of codec system 200 and/or implement steps 101, 103, 105, 107 and/or 109 of method 100 of operation. Encoder 300 splits the input video signal, resulting in split video 301, which is substantially the same as split video 201. Split video 301 is then compressed and encoded into a bitstream by components of encoder 300.

[0088] Specifically, the divided video signal 301 is sent to the intra-picture prediction component 317 for intra prediction. The intra-image prediction component 317 may be substantially similar to the intra-image estimation component 215 and the intra-image prediction component 217 . The divided video signal 301 is also sent to motion compensation component 321 for inter-frame prediction based on reference blocks in decoded image buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219 . The prediction blocks and residual blocks from the intra-image prediction component 317 and the motion compensation component 321 are sent to the transform and quantize component 313 to transform and quantize the residual blocks. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213 . The transformed and quantized residual blocks and corresponding prediction blocks (along with associated control data) are sent to entropy encoding component 331 for encoding into a bitstream. The entropy encoding component 331 may be substantially similar to the header formatting component 231 and CABAC.

[0089] The transformed and quantized residual blocks and/or corresponding prediction blocks are also sent from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into reference blocks for use by the motion compensation component 321. The demapping and quantization component 329 may be substantially similar to the scaling and demapping component 229 . The in-loop filters in the in-loop filter component 325 are also applied to the residual blocks and/or recovered reference blocks, depending on the example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225 . The in-loop filter component 325 may include a plurality of filters, as discussed with respect to the in-loop filter component 225. The filtered blocks are then stored in the decoded image buffer component 323 for use as reference blocks by the motion compensation component 321 . The decoded image buffer component 323 may be substantially similar to the decoded image buffer component 223.

[0090] FIG. 4 is a block diagram illustrating an exemplary video decoder 400. The video decoder 400 may be used to implement decoding functions of the codec system 200 and/or implement steps 111, 113, 115 and/or 117 of the operating method 100. The decoder 400 receives a bit stream, for example, from encoder 300, and generates a reconstructed video output based on the bitstream for display to the end user.

[0091] The bitstream is received by entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme such as CAVLC, CABAC, SBAC, PIPE encoding, or other entropy encoding methods. For example, entropy decoding component 433 may use header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding a video signal, such as general control data, filter control data, section information, motion data, prediction data, and quantized transform coefficients from residual blocks. The quantized transform coefficients are sent to the inverse transform and quantization component 429 for reconstruction into residual blocks. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329 .

[0092] The reconstructed residual blocks and/or prediction blocks are sent to the intra-image prediction component 417 to be reconstructed into image blocks based on intra-prediction operations. The intra-image prediction component 417 may be similar to the intra-image estimation component 215 and the intra-image prediction component 217 . Specifically, the intra-image prediction component 417 uses prediction modes to find a reference block in a frame, and applies a residual block to the result to recover image blocks predicted within the image. The reconstructed intra-prediction image blocks and/or residual blocks and corresponding inter-prediction data are sent to the decoded image buffer component 423 through the in-loop filter component 425, which may be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image blocks, residual blocks, and/or prediction blocks, and such information is stored in the decoded image buffer component 423. The reconstructed image blocks from the decoded image buffer component 423 are sent to the motion compensation component 421 for inter-frame prediction. Motion compensation component 421 may be substantially similar to motion estimation component 221 and/or motion compensation component 219 . Specifically, motion compensation component 421 uses motion vectors from the reference block to form a prediction block and applies the residual block to the result to reconstruct the image block. The resulting reconstructed blocks may also be sent through the in-loop filter component 425 to the decoded image buffer component 423. The decoded image buffer component 423 continues to store additional reconstructed image blocks that can be converted into frames via partition information. Such frames can also be placed sequentially. The sequence is displayed as a reconstructed video output.

[0093] With the above in mind, video compression techniques perform spatial (intra-image) prediction and/or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video sequences. For block video encoding, a video slice (i.e. a video image or part of a video image) can be divided into video blocks, which can also be called tree blocks, coding tree blocks (CTB), coding tree units (CTU), coding units (CU) and/or nodes coding. Video blocks in an intra-encoded (I) slice of an image are encoded using spatial prediction relative to reference samples in adjacent blocks in the same image. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction relative to reference samples in adjacent blocks in the same picture or temporal prediction relative to reference samples in other reference pictures. The images may be called frames, and the reference pictures may be called reference frames.

[0094] Spatial or temporal prediction results in a block prediction for the block to be encoded. The residual data represents the pixel differences between the original block to be encoded and the prediction block. The inter-coded block is encoded according to a motion vector that indicates a block of reference samples forming a prediction block, and residual data indicating the difference between the coded block and the prediction block. The intra-encoded block is encoded according to the intra-encoding mode and residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially located in a two-dimensional array, can be scanned to create a one-dimensional vector of transform coefficients, and entropy encoding can be applied to achieve even greater compression.

[0095] Image and video compression is growing rapidly, leading to the emergence of various encoding standards. These video coding standards include ITU-T H.261, International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) MPEG-1 Part 2, ITU-T H.262 or ISO/IEC MPEG-2 Part 2, ITU- T H.263, ISO/IEC MPEG-4 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.264 or ISO/IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), and Multiview Video Coding plus depth (MVC+D, Multiview Video Coding plus Depth) and 3D AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC) and 3D HEVC (3D-HEVC).

[0096] There is also a new video coding standard called Versatile Video Coding (VVC), which is being developed by the Joint Video Experts Team (JVET) of ITU-T and ISO/IEC. Although the VVC standard has several working drafts, one VVC working draft (WD) in particular, namely B. Bross, J. Chen, and S. Liu, “Versatile Video Coding (Draft 5),” JVET-N1001-v3, 13th JVET Meeting, March 27, 2019 (VVC Draft 5). Each of the references in this paragraph and the preceding paragraph is incorporated by reference in its entirety.

[0097] The description of the methods disclosed in this document is based on the Versatile Video Coding (VVC) video coding standard being developed by the Joint Video Experts Team (JVET) of ITU-T and ISO/IEC. However, these techniques are also applicable to other video codec specifications.

[0098] FIG. 5 is a representation 500 of the relationship between an intra random access point (IRAP) image 502 with respect to leading images 504 and trailing images 506 in decoding order 508 and presentation order 510. In an embodiment, the IRAP picture 502 is referred to as a clean random access (CRA) picture or an instantaneous decoder refresh (IDR) picture with a random access decodable (RADL) picture. In HEVC, IDR images, CRA images , and Broken Link Access (BLA) images are considered 502 IRAP images. Regarding VVC, during the 12th JVET meeting in October 2018, it was decided to use IDR and CRA images as IRAP images. In an embodiment, broken connection access (BLA) and Gradual Decoder Refresh (GDR) pictures may also be considered as IRAP pictures. The decoding process of an encoded video sequence always starts with the IRAP.

[0099] As shown in FIG. 5, leading pictures 504 (eg, pictures 2 and 3) follow the IRAP picture 502 in decoding order 508, but precede the IRAP picture 502 in presentation order 510. The final picture 506 follows the IRAP picture 502 in both decoding order 508 and presentation order 510. Although two leading images 504 and one trailing image 506 are shown in FIG. 5, those skilled in the art will appreciate that more or less leading images 504 and/or trailing images 506 may be present in decoding order 508 and presentation order 510 in practical applications.

[0100] Leading images 504 in FIG. 5 have been divided into two types, namely random access skipped leader (RASL) and RADL. When decoding starts from the IRAP picture 502 (eg, picture 1), the RADL picture (eg, picture 3) can be decoded correctly; however, the RASL picture (eg picture 2) cannot be decoded correctly. Thus, the RASL image is discarded. In light of the difference between RADL and RASL pictures, the type of the leading picture 504 associated with the IRAP picture 502 must be identified as RADL or RASL for efficient and correct encoding. In HEVC, when RASL and RADL pictures are present, it is limited that for RASL and RADL pictures that are associated with the same IRAP picture 502, the RASL pictures must precede the RADL pictures in presentation order 510.

[0101] IRAP Image 502 provides the following two important features/benefits. First, the presence of an IRAP picture 502 indicates that the decoding process can begin with that picture. This functionality allows the use of a random access function in which the decoding process begins at this position in the bitstream, not necessarily at the beginning of the bitstream, as long as the IRAP image 502 is present at that position. Second, the presence of IRAP picture 502 updates the decoding process so that the encoded picture starting with IRAP picture 502, excluding RASL pictures, is encoded without any reference to previous pictures. Having the IRAP picture 502 present in the bitstream will therefore stop any error that may occur during decoding of the encoded pictures up to the IRAP picture 502 from being propagated to the IRAP picture 502 and those pictures that follow the IRAP picture 502 in the decoding order 508.

[0102] Although IRAP images 502 provide important functionality, they reduce compression efficiency. The presence of a 502 IRAP image causes the data rate to spike. This decrease in compression efficiency is due to two reasons. First, since IRAP picture 502 is an intra-prediction picture, the picture itself will require relatively more bits to represent compared to other pictures (eg, leading pictures 504, trailing pictures 506) that are inter-picture prediction pictures. Secondly, since the presence of a 502 IRAP picture violates temporal prediction (this is because the decoder updates the decoding process, in which one of the actions of the decoding process is to remove previous reference pictures in the decoded picture buffer (DPB)) the 502 IRAP picture causes The encoding of pictures following IRAP picture 502 in decoding order 508 becomes less efficient (ie, more bits are required to represent) because they have fewer reference pictures to encode with inter-frame prediction.

[0103] Among the picture types that are considered IRAP pictures 502, the IDR picture in HEVC has a different signaling and origin compared to other picture types. Some differences are as follows.

[0104] To signal and output the picture order count value of an IDR picture, the most significant bit portion of the POC is not derived from the previous key picture, but is simply set to 0.

[0105] With respect to the signaling information necessary to control the reference picture, the IDR picture slice header does not contain information necessary to carry signals to assist in controlling the reference picture. For other picture types (eg, CRA, trailing, TSA, temporal sub-layer access, etc.) Information such as the reference picture set described below, or others forms of similar information (eg, lists of reference pictures) are necessary for the process of marking reference pictures (ie, the process of determining the status of reference pictures in the decoded picture buffer (DPB) that are used for reference or not used for reference). However, for an IDR picture, such information does not need to be reported because the presence of the IDR indicates that the decoding process should simply mark all reference pictures in the DPB as unused for reference.

[0106] In addition to the concept of IRAP images, there are also leading images that, if present, are associated with the IRAP image. Lead pictures are pictures that follow its associated IRAP picture in decoding order, but precede the IRAP picture in output order. Depending on the encoding configuration and image link structure, leading images are further identified into two types. The first type is seed pictures, which may not be decoded correctly if the decoding process starts from the associated IRAP picture. This may occur because these leading pictures are encoded with reference to pictures that precede the IRAP picture in the decoding order. These leading images are called random access skipped leading (RASL) frames. The second type are seed pictures, which must be decoded correctly even if the decoding process starts from the associated IRAP picture. This is possible because these leading pictures are encoded without reference directly or indirectly to pictures that precede the IRAP picture in the decoding order. Such leading pictures are called random access decodable leading pictures (RADL, called random access decodable leading). In HEVC, when RASL and RADL images are present, the constraint is that for RASL and RADL images that are associated with the same IRAP image, the RASL images must precede the RADL images in output order.

[0107] In HEVC and VVC, IRAP images 502 and seed images 504 may each be contained in a single network abstraction layer (NAL) block. A set of NAL blocks may be called an access block. IRAP pictures 502 and seed pictures 504 are assigned different NAL block types so that they can be easily identified by system-level applications. For example, a video editor needs to understand the types of encoded pictures without needing to understand too many details of the syntax element in the encoded bitstream, particularly to identify IRAP pictures 502 from non-IRAP pictures and to identify leading pictures 504, including the definition of RASL and Pictures RADL from trailing pictures 506. Trailing pictures 506 are those pictures that are associated with IRAP picture 502 and follow IRAP picture 502 in presentation order 510. An image may follow a particular IRAP image 502 in decoding order 508 and precede any other IRAP image 502 in decoding order 508. To this end, providing IRAP images 502 and first images 504 of their own NAL block type helps such applications.

[0108] For HEVC, NAL block types for IRAP images include the following:

BLA with Leading Picture (BLA_W_LP): A Broken Link Access (BLA) picture NAL block that may be followed by one or more leading pictures in decoding order.

BLA with RADL (BLA_W_RADL): A NAL block of a BLA picture that may be followed by one or more RADL pictures, but not a RASL picture in decoding order.

BLA without leading picture (BLA_N_LP): A NAL block of a BLA picture that is not followed by a leading picture in decoding order.

IDR with RADL (IDR_W_RADL): A NAL block of an IDR picture that may be followed by one or more RADL pictures, but not a RASL picture in decoding order.

IDR without leading picture (IDR_N_LP): A NAL block of an IDR picture that is not followed by a leading picture in decoding order.

CRA: A NAL block of a Clean Random Access (CRA) image, which may be followed by initial images (ie, either RASL images, RADL images, or both).

RADL: NAL block of RADL image.

RASL: RASL image NAL block.

[0109] For VVC, the NAL block type for IRAP pictures 502 and leading pictures 504 is as follows:

IDR with RADL (IDR_W_RADL): A NAL block of an IDR picture that may be followed by one or more RADL pictures, but not a RASL picture in decoding order.

IDR without leading picture (IDR_N_LP): A NAL block of an IDR picture that is not followed by a leading picture in decoding order.

CRA: A NAL block of a Clean Random Access (CRA) image, which may be followed by initial images (ie, either RASL images or RADL images, or both).

RADL: NAL block of RADL image.

RASL: RASL image NAL block.

[0110] Reference picture resampling (RPR) is the ability to change the spatial resolution of encoded images in the middle of a bitstream without the need for internal image encoding at the location of the resolution change. To make this possible, the image must be able to reference, for inter-picture prediction purposes, one or more reference images for which the spatial resolution is different from the resolution of the current image. Therefore, resampling of such a reference picture or part thereof is necessary for encoding and decoding the current picture. Thus the name RPR. This feature may also be called adaptive resolution rescanning (ARC) or other names. There are use cases or application scenarios for which the RPR feature may be useful, including the following.

[0111] Rate adaptation in video telephony and conferencing. This is necessary to adapt the encoded video to changing network conditions. When network conditions deteriorate and the available bandwidth becomes lower, the encoder can adapt to this by encoding images at a lower resolution.

[0112] Changing the active speaker in a multiparty video conference. For multi-party video conferencing, it is common for the video size for the active speaker to be larger or larger than the video size for the rest of the conference participants. When changing the active speaker, you may also need to adjust the image resolution for each participant. The need to have ARC functionality becomes more important when the active speaker changes frequently.

[0113] A quick start to streaming. For a streaming application, it is common to buffer decoded images to a certain length before starting to display the images. Running the bitstream at a lower resolution will allow the application to have enough images in the buffer for faster rendering.

[0114] Adaptive stream switching in streaming. The Dynamic Adaptive Streaming over HTTP (DASH) specification includes a function named @mediaStreamStructureId. This feature allows you to switch between different views at open-group of picture (GOP) random access points with non-decodable leading pictures, such as CRA pictures with associated RASL pictures in HEVC. When two different representations of the same video have different bitrates but the same spatial resolution and the same @mediaStreamStructureId value, switching between the two representations in a CRA picture with associated RASL pictures can be performed, and the RASL pictures associated with the switching CRA pictures can be decoded with acceptable quality, which ensures smooth switching. With ARC, the @mediaStreamStructureId function can also be used to switch between DASH representations at different spatial resolutions.

[0115] Various techniques simplify basic RPR/ARC support methods, such as signaling of picture resolution lists, some restrictions on resampling of reference pictures in DPB, etc.

[0116] One component of the method required to support RPR is a method for signaling the image resolution that may be present in the bitstream. Some examples solve this by changing the current image resolution signaling with a list of image resolutions in the SPS, as shown below.

seq_parameter_set_rbsp() { Descriptor ... num_pic_size_in_luma_samples_minus1 ue(v) for( i = 0; i <= num_pic_size_in_luma_samples_minus1; i++ ) { pic_width_in_luma_samples [i] ue(v) pic_height_in_luma_samples [i] ue(v) } ... }

[0117] num_pic_size_in_luma_samples_minus1 plus 1 specifies the number of picture sizes (width and height) in units of luma samples that may be present in the encoded video sequence.

[0118] pic_width_in_luma_samples [i] specifies the i-th width of the decoded images in units of luma samples that may be present in the encoded video sequence. pic_width_in_luma_samples [i] must not be 0 and must be an integer multiple of MinCbSizeY.

[0119] pic_height_in_luma_samples [i] specifies the i-th height of the decoded images in units of luminance samples that may be present in the encoded video sequence. pic_height_in_luma_samples [i] must not be 0 and must be an integer multiple of MinCbSizeY.

[0120] During the 15th JVET meeting, another option for signal image size and matching windows to support RPR was discussed. The alarm is as follows.

[0121] - Specify the maximum image size (i.e. image width and height) in SPS

[0122] - Size of the signal image in the picture parameter set (PPS, picture parameter set).

[0123] - Move current matching window signaling from SPS to PPS. The match window information is used to crop the reconstructed/decoded images in the process of preparing the image for output. The cropped image size is the size of the image after cropping the image using its associated match window.

[0124] The image size and matching window signaling is as follows.

seq_parameter_set_rbsp() { Descriptor ... pic_width_in_luma_samples ue(v) pic_height_in_luma_samples ue(v) max_width_in_luma_samples ue(v) max_height_in_luma_samples ue(v) ... }

[0125] max_width_in_luma_samples specifies that the bitstream matching requirement that pic_width_in_luma_samples for any image for which this SPS is active is less than or equal to max_width_in_luma_samples.

[0126] max_height_in_luma_samples specifies that the bitstream matching requirement that pic_height_in_luma_samples for any image for which this SPS is active is less than or equal to max_height_in_luma_samples.

pic_parameter_set_rbsp() { Descriptor ... pps_seq_parameter_set_id ue(v) pic_width_in_luma_samples ue(v) pic_height_in_luma_samples ue(v) conformance_window_flag u(1) if( conformance_window_flag ) { conf_win_left_offset ue(v) conf_win_right_offset ue(v) conf_win_top_offset ue(v) conf_win_bottom_offset ue(v) } ... }

[0127] pic_width_in_luma_samples specifies the width of each decoded image with reference to PPS in units of luma samples. pic_width_in_luma_samples must not be 0 and must be an integer multiple of MinCbSizeY.

[0128] pic_height_in_luma_samples specifies the height of each decoded image with reference to PPS in units of luma samples. pic_height_in_luma_samples must not be 0 and must be an integer multiple of MinCbSizeY.

[0129] The bitstream matching requirement is that all of the following conditions are met for each active reference picture whose width and height are reference_pic_width_in_luma_samples and reference_pic_height_in_luma_samples:

2 * pic_width_in_luma_samples>=reference_pic_width_in_luma_samples

2 * pic_height_in_luma_samples>=reference_pic_height_in_luma_samples

pic_width_in_luma_samples <= 8 * reference_pic_width_in_luma_samples

pic_height_in_luma_samples <= 8 * reference_pic_height_in_luma_samples

[0130] The variables PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC and PicHeightInSamplesC are output as follows:

PicWidthInCtbsY=Ceil (pic_width_in_luma_samples ÷ CtbSizeY) (1)

PicHeightInCtbsY=Ceil (pic_height_in_luma_samples ÷ CtbSizeY) (2)

PicSizeInCtbsY=PicWidthInCtbsY * PicHeightInCtbsY (3)

PicWidthInMinCbsY=pic_width_in_luma_samples/MinCbSizeY (4)

PicHeightInMinCbsY=pic_height_in_luma_samples/MinCbSizeY (5)

PicSizeInMinCbsY=PicWidthInMinCbsY * PicHeightInMinCbsY (6)

PicSizeInSamplesY=pic_width_in_luma_samples * pic_height_in_luma_samples (7)

PicWidthInSamplesC=pic_width_in_luma_samples/SubWidthC (8)

PicHeightInSamplesC=pic_height_in_luma_samples/SubHeightC (9)

[0131] conformance_window_flag equal to 1 indicates that the conformance trim window offset parameters follow further in the PPS. conformance_window_flag equal to 0 indicates that there are no conformance crop window offset options.

[0132] conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset define samples of PPS-referencing images that are output from the decoding process in terms of a rectangular area specified in the output image coordinates. When conformance_window_flag is 0, the values of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offset are considered to be 0.

[0133] The match trim window contains luminance samples with horizontal image coordinates from SubWidthC * conf_win_left_offset to pic_width_in_luma_samples - (SubWidthC * conf_win_right_offset+1) and vertical image coordinates from SubHeightC * conf_win_top_offset to pic_height_in_luma_samples − ( SubHeightC * conf_win_bot tom_offset + 1 ) inclusive.

[0134] The value of SubWidthC * (conf_win_left_offset+conf_win_right_offset) must be less than pic_width_in_luma_samples, and the value of SubHeightC * (conf_win_top_offset+conf_win_bottom_offset) must be less than pic_height_in_luma_samples.

[0135] The PicOutputWidthL and PicOutputHeightL variables are output as follows:

PicOutputWidthL=pic_width_in_luma_samples - (10) SubWidthC * (conf_win_right_offset+conf_win_left_offset)

PicOutputHeightL=pic_height_in_pic_size_units - (11) SubHeightC * (conf_win_bottom_offset+conf_win_top_offset)

[0136] When ChromaArrayType is not 0, the corresponding defined samples of the two chroma arrays are samples having image coordinates (x/SubWidthC, y/SubHeightC), where (x, y) are the image coordinates of the defined luminance samples.

[0137] NOTE - The match trim window offset parameters apply only on output. All internal decoding processes are applied to the uncropped image size.

[0138] Image size and matching window signaling in PPS creates the following problems.

[0139] - Since multiple PPS may be present in a coded video sequence (CVS), it is possible that two PPS may contain the same picture size signaling but different matching windows. This will lead to a situation where two images belonging to different PPS will have the same image size, but different crop size.

[0140] - To support RPR, it has been proposed to disable several encoding tools for block encoding when the current image and the block reference image have different image sizes. However, since it is now possible that the crop size can also be different even if two images have the same image size, it is necessary to have an additional check based on the crop size.

[0141] Disclosed herein are methods that constrain sets of image parameters that have the same image size so that they also have the same matching window size (eg, crop window size). By keeping the matching window the same size for sets of image parameters that have the same image size, excessively complex processing can be avoided when reference picture resampling (RPR) is enabled. In this way, the use of processor, memory and/or network resources can be reduced in both the encoder and decoder. Thus, the encoder/decoder (also known as "codec") in video encoding is improved over current codecs. In practice, an improved video encoding process offers the user a better user experience when sending, receiving and/or watching videos.

[0142] Scalability in video encoding is typically supported through the use of multi-layer encoding techniques. A multi-level bitstream contains a base layer (BL, base layer) and one or more enhancement layers (EL, enhancement layers). Examples of scalability include spatial scalability, quality/signal-to-noise ratio (SNR) scalability, multi-view scalability, etc. When a multi-level coding method is used, an image or part thereof may be encoded (1) without using a reference image, that is, using intra prediction; (2) by referring to reference images that are at the same level, i.e. Using interframe prediction; or (3) by referring to reference images that are on a different level(s), i.e. using cross-layer prediction. The reference picture used for inter-layer prediction of the current image is called an inter-layer reference picture (ILRP, inter-layer reference picture).

[0143] FIG. 6 is a schematic diagram illustrating an example of layer-based prediction 600, for example, performed to determine the MV in the block compression step 105, block decoding step 113, motion estimation component 221, motion compensation component 219, motion compensation component 321, and/or Motion Compensation Component 421. Layer-based prediction 600 is compatible with unidirectional inter-prediction and/or bi-directional inter-prediction, but is also performed between pictures at different layers.

[0144] Level-based prediction 600 is applied between images 611, 612, 613, and 614 and images 615, 616, 617, and 618 at different levels. In the example shown, images 611, 612, 613 and 614 are part of layer N+1 632, and images 615, 616, 617 and 618 are part of layer N 631. A layer such as layer N 631 and/or layer N+1 632, is a group of images that are all associated with a similar characteristic value, such as similar size, quality, resolution, signal-to-noise ratio, capabilities, etc. In the example shown, level N+1,632 is associated with a larger image size than level N 631. Accordingly, images 611, 612, 613, and 614 in level N+1,632 have a larger image size (e.g., greater height and width, and therefore more samples) than images 615, 616, 617 and 618 in level N 631 in this example. However, such images may be separated between the N+1 632 level and the N 631 level by other characteristics. Although only two levels are shown, level N+1 632 and level N 631, the set of images can be divided into any number of levels based on associated characteristics. Level N+1 632 and level N 631 can also be designated by a level identifier. A layer identifier is a data element that is associated with an image and indicates that the image is part of the specified layer. Accordingly, each image 611-618 may be associated with a corresponding layer identifier to indicate which layer N+1 632 or layer N 631 includes the corresponding image.

[0145] Images 611-618 at different levels 631-632 are configured to be displayed in the alternative. As such, images 611-618 at different layers 631-632 may share the same temporary identifier (ID) and may be included in the same AU. As used here, an AU is a set of one or more encoded pictures associated with the same display time for output from the DPB. For example, the decoder may decode and display image 615 at the current display time if a smaller image is required, or the decoder may decode and display image 611 at the current display time if a larger image is required. Essentially, the images 611-614 at the higher layer N+1 632 contain substantially the same image data as the corresponding images 615-618 at the lower layer N 631 (despite the difference in image size). Specifically, image 611 contains substantially the same image data as image 615, image 612 contains substantially the same image data as image 616, etc.

[0146] Pictures 611-618 may be encoded by reference to other pictures 611-618 at the same layer N 631 or N+1 632. Encoding a picture with respect to another picture at the same layer results in inter-frame prediction 623 that is compatible with unidirectional interframe prediction and/or bidirectional interframe prediction. Inter-frame prediction 623 is shown by arrows, solid lines. For example, picture 613 may be encoded using inter-picture prediction 623 using one or two pictures 611, 612 and/or 614 at the N+1 632 level as a reference, where one picture is referenced for unidirectional inter-picture prediction and/or two pictures are reference for bidirectional interframe prediction. In addition, picture 617 may be encoded using inter-picture prediction 623 using one or two pictures 615, 616 and/or 618 at layer N 631 as a reference, where one picture is referenced for unidirectional inter-picture prediction and/or two pictures are reference for bidirectional interframe prediction. When an image is used as a reference for another image at the same level when performing inter-frame prediction 623, the image may be called a reference image. For example, picture 612 may be a reference picture used to encode picture 613 according to inter-picture prediction 623. Inter-picture prediction 623 may also be referred to as intra-layer prediction in a multi-layer context. Thus, inter-picture prediction 623 is a mechanism for encoding samples of a current picture by reference to said samples in a reference picture that is different from the current picture, where the reference picture and the current picture are at the same level.

[0147] Images 611-618 may also be encoded with reference to other images 611-618 at different levels. This process is known as cross-layer prediction 621 and is indicated by dotted arrows. Inter-layer prediction 621 is a mechanism for encoding samples of the current picture with reference to said samples in a reference picture, where the current picture and the reference picture are at different layers and therefore have different layer identifiers. For example, an image in a lower layer N+1 631 may be used as a reference image to encode a corresponding image in a higher layer N+1 632. As a specific example, an image 611 may be encoded with reference to an image 615 according to inter-layer prediction 621. In such a case, the image 615 is used as an inter-layer reference image. An inter-layer reference picture is a reference picture used for inter-layer prediction 621. In most cases, inter-layer prediction 621 is limited such that the current picture, such as picture 611, can only use the inter-layer reference picture(s) that are included into the same AU and are at a lower layer, such as picture 615. When multiple layers are available (eg, more than two), inter-layer prediction 621 may encode/decode the current picture based on multiple cross-layer reference pictures at lower layers than the current image.

[0148] The video encoder may use layer-based prediction 600 to encode pictures 611-618 through a variety of different combinations and/or permutations of inter-frame prediction 623 and inter-layer prediction 621. For example, picture 615 may be encoded according to intra prediction. Pictures 616-618 may then be encoded according to inter-frame prediction 623 using picture 615 as a reference picture. In addition, image 611 may be encoded according to inter-layer prediction 621 using image 615 as an inter-layer reference image. Pictures 612-614 may then be encoded according to inter-picture prediction 623 using picture 611 as a reference picture. As such, a reference picture can serve as either a single-layer reference picture or an inter-layer reference picture for various encoding mechanisms. By encoding N+1,632 higher layer pictures based on lower layer N+1,631 pictures, the higher layer N+1,632 can avoid using intra prediction, which has a much lower coding efficiency than inter-picture prediction 623 and inter-layer prediction 621. Essentially ,low efficiency of intra prediction coding may be limited ,to the smallest/lowest quality images and therefore limited to ,encoding the smallest amount of video data. Pictures used as reference pictures and/or inter-layer reference pictures may be specified in reference picture list(s) entries contained in a reference picture list structure.

[0149] Previous H.26x video encoding families provided scalability support in profile(s) other than the profile(s) for single-layer encoding. Scalable Video Coding (SVC) is a scalable extension to AVC/H.264 that provides support for spatial, temporal, and quality scalability. For the SVC, a flag is signaled in each macroblock (MB) in the EL pictures to indicate whether the EL MB is predicted using a co-located block from the lower layer. The prediction from a co-located block may include texture, motion vectors, and/or encoding modes. SVC implementations cannot directly reuse unmodified H.264/AVC implementations in their projects. The syntax and decoding process for an SVC EL macroblock is different from the syntax and decoding process for H.264/AVC.

[0150] Scalable HEVC (SHVC) is an extension of the HEVC/H.265 standard that provides support for spatial scalability and quality, multi-view HEVC (MV-HEVC) is an extension of HEVC/H.265 that provides support for multi-view scaling, and 3D HEVC (3D-HEVC) is an extension of HEVC/H.264 that provides support for three-dimensional (3D) video encoding, which is more advanced and more efficient than MV-HEVC. Note that temporal scalability is included as an integral part of the single-layer HEVC codec. The HEVC multi-layer extension design uses the idea that the decoded pictures used for inter-layer prediction come only from the same access unit (AU) and are treated as long-term reference pictures (LTRP), and they are assigned reference indices in the reference picture list(s) along with other temporary reference pictures in the current layer. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting a reference index value to reference the inter-layer reference picture(s) in the reference picture list(s).

[0151] Notably, the functions of reference image resampling and spatial scalability require resampling of a reference image or a portion thereof. Resampling of the reference image can be implemented either at the image level or at the encoding block level. However, when RPR is referred to as a coding function, it is a function for single-level coding. Even so, it is possible or even preferable from a codec design point of view to use the same oversampling filter for both the single-level RPR function and the multi-level spatial scalability function.

[0152] FIG. 7 is a schematic diagram illustrating an example of unidirectional inter-frame prediction 700. Uni-directional inter-frame prediction may be used to determine motion vectors for coded and/or decoded blocks created by dividing an image.

[0153] Unidirectional inter-frame prediction 700 uses a reference frame 730 with a reference block 731 to predict the current block 711 in the current frame 710. The reference frame 730 may be placed in time after the current frame 710 as shown (eg, as a subsequent reference frame), but may also be temporarily located before the current frame 710 (eg, as a previous reference frame) in some examples. The current frame 710 is an example of a frame/image being encoded/decoded at a particular time. Current frame 710 contains an object in current block 711 that corresponds to an object in reference block 731 of reference frame 730. Reference frame 730 is a frame that is used as a reference for encoding the current frame 710, and reference block 731 is a block in reference frame 730 , which contains an object also contained in the current block 711 of the current frame 710.

[0154] Current block 711 is any encoding unit that is encoded/decoded at a specific point in the encoding process. The current block 711 may be an entire divided block or may be a subblock when using the affine inter-prediction mode. The current frame 710 is separated from the reference frame 730 by a temporal distance (TD) 733. TD 733 indicates the amount of time between the current frame 710 and the reference frame 730 in the video sequence and can be measured in units of frames. The prediction information for the current block 711 may reference the reference frame 730 and/or the reference block 731 by means of a reference index indicating the direction and temporal distance between frames. During the period of time represented by TD 733, an object in the current block 711 moves from a position in the current frame 710 to another position in the reference frame 730 (eg, the position of the reference block 731). For example, an object may move along a motion path 713, which is the direction in which the object moves over time. Motion vector 735 describes the direction and magnitude of motion of an object along motion path 713 over TD 733. Accordingly, encoded motion vector 735, reference block 731, and a remainder including the difference between current block 711 and reference block 731 provide information sufficient to reconstruct current block 711 and placing the current block 711 in the current frame 710.

[0155] FIG. 8 is a schematic diagram illustrating an example of bidirectional inter-prediction 800. Bi-directional inter-prediction may be used to determine motion vectors for coded and/or decoded blocks created by dividing an image.

[0156] Bidirectional interframe prediction 800 is similar to unidirectional interframe prediction 700, but uses a pair of reference frames to predict the current block 811 in the current frame 810. Therefore, the current frame 810 and the current block 811 are essentially the same as the current frame 710 and the current block 711, respectively. The current frame 810 is positioned in time between a previous reference frame 820 that occurs before the current frame 810 in the video sequence and a subsequent reference frame 830 that occurs after the current frame 810 in the video sequence. The previous reference frame 820 and the subsequent reference frame 830 are otherwise substantially the same as the reference frame 730.

[0157] The current block 811 corresponds to the previous reference block 821 in the previous reference frame 820 and the subsequent reference block 831 in the subsequent reference frame 830. Such a match indicates that during the video sequence the object moves from a position in the previous reference block 821 to a position in the subsequent reference block 831 along the motion path 813 and through the current block 811. The current frame 810 is separated from the previous reference frame 820 by some previous time distance (TD0) 823 and is separated from the subsequent reference frame 830 by some subsequent time distance (TD1) 833. TD0 823 indicates the amount of time between the previous reference frame 820 and the current frame 810 in the video sequence in frame units. TD1 833 indicates the amount of time between the current frame 810 and the subsequent reference frame 830 in the video sequence in frame units. Therefore, the object moves from the previous reference block 821 to the current block 811 along the motion path 813 during the period of time indicated by TD0 823. The object also moves from the current block 811 to the subsequent reference block 831 along the motion path 813 during the time period indicated by TD1 833 The prediction information for the current block 811 may reference a previous reference frame 820 and/or a previous reference block 821 and a subsequent reference frame 830 and/or a subsequent reference block 831 by a pair of reference indices indicating the direction and temporal distance between frames.

[0158] Previous motion vector (MV0) 825 describes the direction and amount of movement of an object along a motion path 813 along TD0 823 (eg, between the previous reference frame 820 and the current frame 810). The subsequent motion vector (MV1) 835 describes the direction and amount of movement of an object along a motion path 813 along TD1 833 (eg, between the current frame 810 and the subsequent reference frame 830). As such, in bidirectional interframe prediction 800, the current block 811 can be encoded and reconstructed using the previous reference block 821 and/or the subsequent reference block 831, MV0 825 and MV1 835.

[0159] In an embodiment, inter-frame prediction and/or bi-directional inter-frame prediction may be performed on a sample-by-sample basis (eg, pixel by pixel) rather than on a block-by-block basis. That is, a motion vector pointing to each sample in the previous reference block 821 and/or the subsequent reference block 831 may be determined for each sample in the current block 811. In such embodiments, motion vector 825 and motion vector 835 shown in FIG. 8 represent a plurality of motion vectors corresponding to a plurality of samples in the current block 811, the previous reference block 821, and the subsequent reference block 831.

[0160] In both fusion mode and advanced motion vector prediction (AMVP) mode, the candidate list is generated by adding candidate motion vectors to the candidate list in the order specified by the candidate list definition template. Such candidate motion vectors may include motion vectors according to unidirectional inter-prediction 700, bi-directional inter-prediction 800, or combinations thereof. In particular, motion vectors are generated for adjacent blocks when such blocks are encoded. Such motion vectors are added to the candidate list for the current block, and the motion vector for the current block is selected from the candidate list. The motion vector can then be passed as the index of the selected motion vector in the candidate list. The decoder can create a candidate list using the same process as the encoder and can determine the selected motion vector from the candidate list based on the passed index. Therefore, the candidate motion vectors include motion vectors generated according to unidirectional inter-prediction 700 and/or bi-directional inter-prediction 800, depending on which approach is used in encoding such adjacent blocks.

[00161] FIG. 9 illustrates a video bitstream 900. In this context, video bitstream 900 may also be referred to as encoded video bitstream, bitstream, or. As shown in FIG. 9, bitstream 900 contains a sequence parameter set 902, a picture parameter set 904, a slice header 906, and picture data 908.

[00162] SPS 902 contains data common to all pictures in a sequence of pictures (SOP, sequence of pictures). In contrast, PPS 904 contains data common to the entire image. The slice header 906 contains information about the current slice, such as, for example, the slice type, which reference image will be used, and so on. SPS 902 and PPS 904 can generally be called a parameter set. SPS 902, PPS 904 and slice header 906 are network abstraction layer (NAL) block types. A NAL block is a syntactic structure containing an indication of the type of data to be followed (for example, encoded video data). NAL blocks are divided into video coding layer (VCL) blocks and non-VCL NAL blocks. VCL NAL blocks contain data that represents sample values in video images, and non-VCL NAL blocks contain any associated additional information such as parameter sets (important header data that can apply to a large number of VCL NAL blocks) and additional enhancement information (timing information and other additional data that may enhance the usability of the decoded video signal, but is not necessary for decoding sample values in video images). Those skilled in the art will appreciate that bitstream 900 may contain other parameters and information in practical applications.

[00163] Image data 908 of FIG. 9 contain data associated with images or videos that are encoded or decoded. The image data 908 may be referred to simply as the payload or data carried in the bitstream 900. In an embodiment, the image data 908 comprises a CVS 914 (or CLVS) containing a plurality of images 910. The CVS 914 is a coded video sequence for each coded layer video sequence (CLVS, coded layer video sequence) in the 900 video bitstream. Notably, CVS and CLVS are the same when the video bitstream includes a single layer. CVS and CLVS differ only when the video bitstream 900 includes multiple layers.

[00164] As shown in FIG. 9, a slice of each image 910 may be contained in its own NAL block 912VCL. A set of VCL NAL blocks 912 in CVS 914 may be referred to as an access block.

[00165] FIG. 10 illustrates a division method 1000 for image 1010. Image 1010 may be similar to any of the images 910 in FIG. 9. As shown, an image 1010 may be divided into a plurality of slices 1012. A slice is a spatially distinct region of a frame (eg, an image) that is encoded separately from any other region in the same frame. Although in FIG. 10 shows three slices 1012, in practical applications more or fewer slices may be used. Each slice 1012 may be divided into a plurality of blocks 1014. Blocks 1014 in FIG. 10 may be similar to current block 811, previous reference block 821, and subsequent reference block 831 in FIG. 8. Block 1014 may represent a CU. Although four blocks 1014 are shown in FIG. 10, in practical applications, more or fewer blocks can be used.

[00166] Each block 1014 may be divided into a plurality of samples 1016 (eg, pixels). In an embodiment, the size of each block 1014 is measured in luminance samples. Although sixteen samples 1016 are depicted in FIG. 10, more or fewer samples can be used in practical applications.

[00167] In an embodiment, a match window 1060 is applied to the image 1010. As noted above, the correspondence window 1060 is used to crop, reduce, or otherwise resize the image 1010 (eg, the reconstructed/decoded image) in the process of preparing the image for output. For example, the decoder may apply a match window 1060 to the image 1010 to crop, adjust, compress, or otherwise resize the image image 1010 before the image is output for display to the user. The size of the match window 1060 is determined by applying a match window up offset 1062, a match window down offset 1064, a match window left offset 1066, and a match window right offset 1068 to the image 1010 to reduce the size of the image 1010 before output. That is, only the portion of the image 1010 that exists in the matching window 1060 is output. Thus, the image 1010 is cropped to size before output. In an embodiment, each first image parameter set and second image parameter set belong to the same sequence parameter set and have the same image width and image height values. Essentially, the first set of image parameters and the second set of image parameters have the same values for the matching window.

[00168] FIG. 11 illustrates an embodiment of a decoding method 1100 implemented by a video decoder (eg, video decoder 400). Method 1100 may be performed after the decoded bitstream has been directly or indirectly received from a video encoder (eg, video encoder 300). Method 1100 improves the decoding process by maintaining the same match window size for image parameter sets having the same image size. Thus, reference picture resampling (RPR) can remain enabled or enabled for the entire CVS. By maintaining a consistent matching window size for sets of image parameters having the same image size, coding efficiency can be improved. Therefore, from a practical point of view, the codec's performance is improved, resulting in a better user experience.

[00169] At block 1102, the video decoder receives a first set of picture parameters (eg, ppsA) and a second set of picture parameters (eg, ppsB), each of which refers to the same set of sequence parameters. When the first set of image parameters and the second set of image parameters have the same image width and height values, the first set of image parameters and the second set of image parameters have the same match window values. In an embodiment, the image width and image height are measured in luminance samples.

[00170] In an embodiment, the width of the image is denoted as pic_width_in_luma_samples. In an embodiment, the height of the image is denoted as pic_height_in_luma_samples. In an embodiment, pic_width_in_luma_samples specifies the width of each decoded PPS image in units of luma samples. In an embodiment, pic_height_in_luma_samples specifies the height of each decoded image with reference to PPS in units of luma samples.

[00171] In an embodiment, the match window comprises a match window left offset, a match window right offset, a match window offset up, and a match window offset down, which together represent the size of the match window. In an embodiment, the left offset of the matching window is denoted as pps_conf_win_left_offset. In an embodiment, the right offset of the matching window is denoted as pps_conf_win_right_offset. In an embodiment, the top offset of the matching window is denoted as pps_conf_win_top_offset. In an embodiment, the downward offset of the matching window is denoted as pps_conf_win_bottom_offset. In an embodiment, the size or values of the matching window are reported in the PPS.

[00172] At block 1104, the video decoder applies a match window to the current image corresponding to the first set of image parameters or the second set of image parameters. Thus, the video encoder crops the current image to the size of the matching window.

[00173] In an embodiment, the method further comprises using inter-frame prediction to decode the current picture based on the resampled reference picture. In an embodiment, the method further comprises resampling a reference picture corresponding to the current picture using reference picture resampling (RPS). In an embodiment, resampling the reference image changes the resolution of the reference image.

[00174] In an embodiment, the method further comprises determining whether bi-direction optical flow (BDOF) is enabled for decoding the image based on an image width, an image height, and a matching window between the current image and a reference image for the current image. In an embodiment, the method further comprises determining whether decoder-side motion vector refinement (DMVR) is enabled for decoding a picture based on a picture width, a picture height, and a matching window between the current picture and a reference picture for the current picture.

[00175] In an embodiment, the method further comprises displaying on the display of an electronic device (eg, a smartphone, tablet, laptop, personal computer, etc.) an image created using the current block.

[00176] FIG. 12 illustrates an embodiment of a method 1200 for encoding a video bitstream implemented by a video encoder (eg, video encoder 300). Method 1200 may be performed when an image (eg, from a video) is to be encoded into a video bitstream and then transmitted to a video decoder (eg, video decoder 400). Method 1200 improves the encoding process by maintaining the same match window size for image parameter sets having the same image size. Thus, reference picture resampling (RPR) can remain enabled or enabled for the entire CVS. By maintaining a consistent matching window size for sets of image parameters having the same image size, coding efficiency can be improved. Therefore, from a practical point of view, the codec's performance is improved, resulting in a better user experience.

[00177] At step 1202, the video encoder generates a first set of image parameters and a second set of image parameters, each of which relates to the same set of sequence parameters. When the first set of image parameters and the second set of image parameters have the same image width and height values, the first set of image parameters and the second set of image parameters have the same match window values. In an embodiment, the image width and image height are measured in luminance samples.

[00178] In an embodiment, the width of the image is denoted as pic_width_in_luma_samples. In an embodiment, the height of the image is denoted by pic_height_in_luma_samples. In an embodiment, pic_width_in_luma_samples specifies the width of each decoded PPS image in units of luma samples. In an embodiment, pic_height_in_luma_samples specifies the height of each decoded image with reference to PPS in units of luma samples.

[00179] In an embodiment, the match window comprises a match window left offset, a match window right offset, a match window offset up, and a match window offset down, which together represent the size of the match window. In an embodiment, the left offset of the matching window is denoted as pps_conf_win_left_offset. In an embodiment, the right offset of the matching window is denoted as pps_conf_win_right_offset. In an embodiment, the top offset of the matching window is denoted as pps_conf_win_top_offset. In an embodiment, the downward offset of the matching window is denoted as pps_conf_win_bottom_offset. In an embodiment, the size or values of the matching window are reported in the PPS.

[00180] At step 1204, the video encoder encodes the first set of image parameters and the second set of image parameters into a video bitstream. At step 1206, the video encoder stores the video bitstream for transmission to the video decoder. In an embodiment, the video encoder transmits a video bitstream containing a first set of image parameters and a second set of image parameters to the video decoder.

[00181] In an embodiment, a method for encoding a video bitstream is provided. The bitstream contains many sets of parameters and many images. Each image of a plurality of images contains a plurality of slices. Each slice of the plurality of slices contains a plurality of encoding blocks. The method comprises generating and writing a set of parameters parameterSetA into a bit stream containing information that includes the image size picSizeA and the matching window confWinA. The parameter can be a set of picture parameters (PPS, picture parameter set). The method further includes creating and writing another parameter set parameterSetB to a bitstream containing information that includes an image size picSizeB and a match window confWinB. The parameter can be a set of picture parameters (PPS, picture parameter set). The method further includes constraining the values for the match window confWinA in parameterSetA and confWinB in parameterSetB to be the same when the values of picSizeA in parameterSetA and picSizeB in parameterSetB are the same, and constraining the image size values picSizeA in parameterSetA and picSizeB in parameterSetB to be the same if the values of confWinA in parameterSetA and confWinB in parameterSetB are the same. The method further includes encoding the bit stream.

[00182] In an embodiment, a method for decoding a video bitstream is provided. The bitstream contains many sets of parameters and many images. Each image of a plurality of images contains a plurality of slices. Each slice of the plurality of slices contains a plurality of encoding blocks. The method comprises parsing a set of parameters to obtain an image size and a matching window size associated with the current image currPic. The information obtained is used to determine the image size and crop size of the current image. The method further includes parsing another set of parameters to obtain an image size and a match window size associated with the reference image refPic. The information obtained is used to determine the image size and crop size of the reference image. The method further includes determining refPic as a reference picture for decoding the current curBlock that is in the current picture currPic, determining whether bi-direction optical flow (BDOF) is used or enabled for decoding the current encoding block based on the image size and the matching window between the current image and the reference image, and decoding the current block.

[00183] In an embodiment, BDOF is not used or disabled for decoding the current encoding block when the image size and correspondence window of the current image and the reference image are different.

[00184] In an embodiment, a method for decoding a video bitstream is provided. The bitstream contains many sets of parameters and many images. Each image of a plurality of images contains a plurality of slices. Each slice of the plurality of slices contains a plurality of encoding blocks. The method comprises parsing a set of parameters to obtain an image size and a matching window size associated with the current image currPic. The information obtained is used to determine the image size and crop size of the current image. The method further includes parsing another set of parameters to obtain an image size and a match window size associated with the reference image refPic. The information obtained is used to determine the image size and crop size of the reference image. The method further includes determining refPic as a reference picture for decoding the current curBlock that is in the current picture currPic, determining whether decoder-side motion vector refinement (DMVR) is used on the decoder side to decode the current encoding block based on the image size and window matching of the current image and the reference image, as well as the decoding of the current block.

[00185] In an embodiment, the DMVR is not used or disabled for decoding the current encoding block when the picture size and matching window of the current picture and the reference picture are different.

[00186] In an embodiment, a method for encoding a video bitstream is provided. In an embodiment, the bitstream contains a plurality of parameter sets and a plurality of images. Each image of a plurality of images contains a plurality of slices. Each slice of the plurality of slices contains a plurality of encoding blocks. The method includes creating a parameter set containing an image size and a match window size associated with the current image currPic. The information is used to obtain the image size and crop size of the current image. The method further comprises creating another set of parameters comprising an image size and a matching window size associated with the reference image refPic. The information obtained is used to determine the image size and crop size of the reference image. The method further includes a constraint that the reference picture refPic should not be used as a combined reference picture for temporal motion vector prediction (TMVP) of all slices that belong to the current picture currPic when the size of the picture and the matching window of the current picture and the reference image is different. That is, the constraint is that if the reference picture refPic is a co-located reference picture for encoding blocks in the current picture currPic for TMVP, the picture size and matching window of the current picture and the reference picture must be the same. The method further includes decoding the bitstream.

[00187] In an embodiment, a method for decoding a video bitstream is provided. The bitstream contains many sets of parameters and many images. Each image of a plurality of images contains a plurality of slices. Each slice of the plurality of slices contains a plurality of encoding blocks. The method comprises parsing a set of parameters to obtain an image size and a matching window size associated with the current image currPic. The information obtained is used to determine the image size and crop size of the current image. The method further includes parsing another set of parameters to obtain an image size and a match window size associated with the reference image refPic. The information obtained is used to determine the image size and crop size of the reference image. The method further includes defining a refPic as a reference image for decoding the current curBlock that is in the current image currPic, parsing the syntax element (slice_DVMR_BDOF_enable_flag) to determine whether decoder-side motion vector refinement (DMVR) is required. on the decoder side and/or bi-direction optical flow (BDOF) is used or enabled to decode the current encoding picture and/or slice. The method further includes limiting the values for the syntax element (slice_DVMR_BDOF_enable_flag) to zero when the matching window confWinA in parameterSetA and confWinB in parameterSetB are not the same or when the values of picSizeA in parameterSetA and picSizeB in parameterSetB are not the same.

[00188] The following description refers to the main text, which is the VVC working draft. That is, only the delta is described, while texts in the main text that are not mentioned below are applied as is. Deleted text appears in italics, and added text appears in bold.

[00189] Sequence parameter set syntax and semantics are provided.

seq_parameter_set_rbsp() { Descriptor ... pic_width_in_luma_samples ue(v) pic_height_in_luma_samples ue(v) max_width_in_luma_samples ue(v) max_height_in_luma_samples ue(v) ... }

[00190] max_width_in_luma_samples specifies that the bitstream matching requirement that pic_width_in_luma_samples for any image for which this SPS is active is less than or equal to max_width_in_luma_samples.

[00191] max_height_in_luma_samples specifies that the bitstream matching requirement that pic_height_in_luma_samples for any image for which this SPS is active is less than or equal to max_height_in_luma_samples.

[00192] The syntax and semantics of an image parameter set are provided.

pic_parameter_set_rbsp() { Descriptor ... pps_seq_parameter_set_id ue(v) pic_width_in_luma_samples ue(v) pic_height_in_luma_samples ue(v) conformance_window_flag u(1) if (conformance_window_flag) { conf_win_left_offset ue(v) conf_win_right_offset ue(v) conf_win_top_offset ue(v) conf_win_bottom_offset ue(v) } ... }

[00193] pic_width_in_luma_samples specifies the width of each decoded image with reference to PPS in units of luma samples. pic_width_in_luma_samples must not be 0 and must be an integer multiple of MinCbSizeY.

[00194] pic_height_in_luma_samples specifies the height of each decoded image with reference to PPS in units of luma samples. pic_height_in_luma_samples must not be 0 and must be an integer multiple of MinCbSizeY.

[00195] The bitstream matching requirement is that all of the following conditions are met for each active reference picture whose width and height are reference_pic_width_in_luma_samples and reference_pic_height_in_luma_samples:

[00196] - 2 * pic_width_in_luma_samples>=reference_pic_width_in_luma_samples

[00197] - 2 * pic_height_in_luma_samples>=reference_pic_height_in_luma_samples

[00198] - pic_width_in_luma_samples <= 8 * reference_pic_width_in_luma_samples

[00199] - pic_height_in_luma_samples <= 8 * reference_pic_height_in_luma_samples

[00200] The variables PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC and PicHeightInSamplesC are output as follows.

[00201] PicWidthInCtbsY=Ceil (pic_width_in_luma_samples ÷ CtbSizeY) (1)

[00202] PicHeightInCtbsY=Ceil (pic_height_in_luma_samples ÷ CtbSizeY) (2)

[00203] PicSizeInCtbsY=PicWidthInCtbsY * PicHeightInCtbsY (3)

[00204] PicWidthInMinCbsY=pic_width_in_luma_samples/MinCbSizeY (4)

[00205] PicHeightInMinCbsY=pic_height_in_luma_samples/MinCbSizeY (5)

[00206] PicSizeInMinCbsY=PicWidthInMinCbsY * PicHeightInMinCbsY (6)

[00207] PicSizeInSamplesY=pic_width_in_luma_samples * pic_height_in_luma_samples (7)

[00208] PicWidthInSamplesC=pic_width_in_luma_samples/SubWidthC (8)

[00209] PicHeightInSamplesC=pic_height_in_luma_samples/SubHeightC (9)

[00210] conformance_window_flag equal to 1 indicates that the conformance trim window offset parameters follow further in the PPS. conformance_window_flag equal to 0 indicates that there are no conformance crop window offset options.

[00211] conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset define samples of PPS-referencing images that are output from the decoding process in terms of a rectangular area specified in the output image coordinates. When conformance_window_flag is 0, the values of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offset are considered to be 0.

[00212] The match trim window contains luminance samples with horizontal image coordinates from SubWidthC * conf_win_left_offset to pic_width_in_luma_samples - (SubWidthC * conf_win_right_offset+1) and vertical image coordinates from SubHeightC * conf_win_top_offset to pic_height_in_luma_samples − ( SubHeightC * conf_win_ bottom_offset + 1 ) inclusive.

[00213] The value of SubWidthC * (conf_win_left_offset+conf_win_right_offset) must be less than pic_width_in_luma_samples, and the value of SubHeightC * (conf_win_top_offset+conf_win_bottom_offset) must be less than pic_height_in_luma_samples.

[00214] The PicOutputWidthL and PicOutputHeightL variables are output as follows:

[00215] PicOutputWidthL=pic_width_in_luma_samples - (10)

[00216] SubWidthC * (conf_win_right_offset+conf_win_left_offset)

[00217] PicOutputHeightL=pic_height_in_pic_size_units - (11)

[00218] SubHeightC * (conf_win_bottom_offset+conf_win_top_offset)

[00219] When ChromaArrayType is not equal to 0, the corresponding defined samples of the two chroma arrays are samples having image coordinates (x/SubWidthC, y/SubHeightC), where (x, y) are the image coordinates of the defined luminance samples.

[00220] NOTE: The match crop window offset settings apply only to output. All internal decoding processes are applied to the uncropped image size.

[00221] Let PPS_A and PPS_B be image parameter sets related to the same sequence parameter set, the bitstream matching requires all of the following conditions to be met if the pic_width_in_luma_samples values in PPS_A and PPS_B are the same and the pic_height_in_luma_samples values in PPS_A and PPS_B are the same:

The conf_win_left_offset values in PPS_A and PPS_B are the same

The conf_win_right_offset values in PPS_A and PPS_B are the same

The conf_win_top_offset values in PPS_A and PPS_B are the same

The conf_win_bottom_offset values in PPS_A and PPS_B are the same

[00222] The following constant is added to the collocated_ref_idx semantics

[00223] collocated_ref_idx defines the reference index of a collocated image used for temporal motion vector prediction.

[00224] When slice_type is P or when slice_type is B and collocated_from_l0_flag is 1, collocated_ref_idx refers to the image in list 0, and the value of collocated_ref_idx must be in the range from 0 to NumRefIdxActive[0] - 1 inclusive. .

[00225] When slice_type is B and collocated_from_l0_flag is 0, collocated_ref_idx refers to the image in list 1, and the value of collocated_ref_idx must be in the range from 0 to NumRefIdxActive[1] - 1 inclusive.

[00226] When collocated_ref_idx is missing, the value of collocated_ref_idx is assumed to be 0.

[00227] The bitstream matching requirement is that the image referenced by collocated_ref_idx be the same across all slices of the encoded image.

[00228] Bitstream matching requires that the resolutions of the reference image referenced by collocated_ref_idx and the current image be the same.

[00229] The bitstream matching requirement is that the dimensions of the image and matching window of the reference image referenced by collocated_ref_idx and the current image are the same.

[00230] The following conditions for setting dmvrFlag to 1 have been changed.

[00231] - When all of the following conditions are met, dmvrFlag is set to 1:

[00232] - sps_dmvr_enabled_flag is equal to 1

[00233] - general_merge_flag [xCb] [yCb] equals 1

[00234] - both predFlagL0 [0] [0] and predFlagL1 [0] [0] are 1

[00235] - mmvd_merge_flag [xCb] [yCb] is 0

[00236] - DiffPicOrderCnt (currPic, RefPicList [0] [refIdxL0]) equals DiffPicOrderCnt (RefPicList [1] [refIdxL1], currPic)

[00237] - BcwIdx [xCb] [yCb] equals 0

[00238] - Both luma_weight_l0_flag [refIdxL0] and luma_weight_l1_flag [refIdxL1] are 0

[00239] - cbWidth is greater than or equal to 8

[00240] - cbHeight is greater than or equal to 8

[00241] - cbHeight * cbWidth greater than or equal to 128

[00242] - - For X equal to 0 and 1, the pic_width_in_luma_samples and pic_height_in_luma_samples of the reference image refPicLX associated with refIdxLX are equal to the pic_width_in_luma_samples and pic_height_in_luma_samples of the current image, respectively.

[00243] - - For X equal to 0 and 1, pic_width_in_luma_samples, pic_height_in_luma_samples, conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offset of the refPicLX reference image associated with refIdxLX are equal to pic_width_in_luma_samples, pic_height_in_luma_samples, conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offset of the current image respectively.

[00244] The following conditions for setting dmvrFlag to 1 have been changed.

[00245] - If all of the following conditions are true, bdofFlag is set to TRUE.

[00246] - sps_bdof_enabled_flag is equal to 1.

[00247] - predFlagL0 [xSbIdx] [ySbIdx] and predFlagL1 [xSbIdx] [ySbIdx] are both 1.

[00248] - DiffPicOrderCnt (currPic, RefPicList [0] [refIdxL0]) * DiffPicOrderCnt (currPic, RefPicList [1] [refIdxL1]) is less than 0.

[00249] - MotionModelIdc [xCb] [yCb] is 0.

[00250] - merge_subblock_flag [xCb] [yCb] is equal to 0.

[00251] - sym_mvd_flag [xCb] [yCb] is equal to 0.

[00252] - BcwIdx [xCb] [yCb] is equal to 0.

[00253] - luma_weight_l0_flag [refIdxL0] and luma_weight_l1_flag [refIdxL1] are both 0.

[00254] - cbHeight is greater than or equal to 8

[00255] - - For X equal to 0 and 1, the pic_width_in_luma_samples and pic_height_in_luma_samples of the reference image refPicLX associated with refIdxLX are equal to the pic_width_in_luma_samples and pic_height_in_luma_samples of the current image, respectively.

[00256] - - For X equal to 0 and 1, pic_width_in_luma_samples, pic_height_in_luma_samples, conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offse of the refPicLX reference image associated with refIdxLX are equal to pic_width_in_luma_samples , pic_height_in_luma_samples, conf_win_left_offset, conf_win_right_offset, conf_win_top_offset and conf_win_bottom_offse of the current image respectively.

[00257] - cIdx is 0.

[00258] FIG. 13 is a schematic diagram of a video encoding device 1300 (eg, video encoder 20 or video decoder 30) according to an embodiment of the invention. Video encoding apparatus 1300 is suitable for implementing the disclosed embodiments that are described herein. The video encoding device 1300 includes input ports 1310 and receiver units (Rx) 1320 for receiving data; processor, logical unit or central processing unit (CPU, central processing unit) 1330 for data processing; transmitter (Tx) blocks 1340 and data output ports 1350; and 1360 memory for data storage. The video encoding device 1300 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components connected to input ports 1310, receiver blocks 1320, blocks 1340 transmitter and output ports 1350 to provide input or output of optical or electrical signals.

[00259] The processor 1330 is implemented in hardware and software. The processor 1330 may be implemented as one or more CPU chips, cores (eg, as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors ( DSP, digital signal processor). Processor 1330 communicates with input ports 1310, receiver blocks 1320, transmitter blocks 1340, output ports 1350, and memory 1360. Processor 1330 includes an encoding module 1370. Encoding module 1370 implements the disclosed embodiments described above. For example, encoding module 1370. Therefore, the inclusion of the encoding module 1370 provides a significant improvement in the functionality of the video encoding device 1300 and allows the video encoding device 1300 to be transformed into a different state. Alternatively, encoding module 1370 is implemented as instructions stored in memory 1360 and executed by processor 1330.

[00260] Video encoding device 1300 may also include input and/or output (I/O) devices 1380 for transmitting data to and from the user. The input/output devices 1380 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. I/O devices 1380 may also include input devices such as a keyboard, mouse, trackball, etc., and/or corresponding interfaces for interacting with such output devices.

[00261] Memory 1360 contains one or more disks, tape drives, and solid state drives and can be used as an overflow storage device, for storing programs when such programs are selected for execution, and for storing instructions and data that are read during execution. programs. Memory 1360 may be volatile and/or non-volatile and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and /or static random-access memory (SRAM, static random-access memory).

[00262] FIG. 14 is a schematic diagram of an embodiment of encoding means 1400. In an embodiment, the encoding means 1400 is implemented in a video encoding device 1402 (eg, video encoder 20 or video decoder 30). The video encoding device 1402 includes a receiving means 1401. The receiving means 1401 is configured to receive an image for encoding or receive a bitstream for decoding. The video encoding device 1402 includes a transmission means 1407 connected to a reception means 1401. The transmission means 1407 is configured to transmit a bit stream to a decoder or transmit a decoded image to a display device (eg, one of the input/output devices 1380).

[00263] Video encoding device 1402 includes storage means 1403. The storage means 1403 is connected to at least one of the receiving means 1401 or the transmitting means 1407. The storage means 1403 is configured to store instructions. Video encoding device 1402 also includes processing means 1405. The processing means 1405 is connected to the storage means 1403. The processing means 1405 is configured to execute instructions stored in the storage means 1403 to implement the methods.

[00264] It should also be understood that the steps of the exemplary methods set forth herein need not be performed in the order described, and the order of the steps of such methods is to be understood as merely exemplary. Likewise, additional steps may be included in such methods, and some steps may be omitted or combined in methods consistent with various embodiments of the present disclosure.

[00265] Although the present disclosure presents several embodiments, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. These examples should be considered illustrative and not restrictive, and it is not intended to be limited by the details given here. For example, various elements or components may be combined or integrated into another system, or certain functions may be omitted or not implemented.

[00266] Additionally, the methods, systems, subsystems, and methods described and illustrated in various embodiments as separate or distinct may be combined or integrated with other systems, modules, technologies, or methods without departing from the scope of the present invention. Other elements shown or discussed as being connected or directly connected or interacting with each other may be indirectly connected or interacting through some interface, device or intermediate component, electrically, mechanically or otherwise. Other examples of changes, substitutions and alterations may be ascertained by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims (47)

1. A decoding method implemented by a video decoder, containing: receiving by a video decoder a first picture parameter set (PPS) and a second picture parameter set, each of which refers to the same sequence parameter set (SPS), wherein each of the first picture parameter set and the second picture parameter set comprises pic_width_in_luma_samples, pic_height_in_luma_samples and parameters matching windows, wherein pic_width_in_luma_samples indicates the width of each decoded image referring to the corresponding set of image parameters, in units of luma samples, and pic_height_in_luma_samples indicates the height of each decoded image referring to the corresponding set of image parameters, in units of luma samples; wherein, when the first set of image parameters and the second set of image parameters have the same values of pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, the first set of image parameters and the second set of image parameters have the same values of the correspondence window parameters; And applying the match window to the current image corresponding to the first set of image parameters or the second set of image parameters by the video decoder. 2. The method according to claim 1, wherein the parameters of the match window comprise a left offset of the match window, a right offset of the match window, an upward offset of the match window, and a downward offset of the match window. 3. Method according to any one of paragraphs. 1, 2, further comprising decoding a current image corresponding to the first set of image parameters or the second set of image parameters using inter-frame prediction after applying a matching window, wherein the inter-frame prediction is based on the resampled reference image. 4. Method according to any one of paragraphs. 1-3, further comprising resampling a reference picture associated with a current picture corresponding to the first set of picture parameters or the second set of picture parameters using reference picture resampling (RPS). 5. The method of claim 4, wherein resampling the reference image changes the resolution of the reference image used for inter-frame prediction of the current image corresponding to the first set of image parameters or the second set of image parameters. 6. Method according to any one of paragraphs. 1-5, further comprising determining whether bidirectional optical flow (BDOF) is enabled for decoding the current picture based on pic_width_in_luma_samples, pic_height_in_luma_samples, and match window parameters of the current picture and a reference picture for the current picture. 7. Method according to any one of paragraphs. 1-5, further comprising determining whether motion vector refinement (DMVR) is enabled on the decoder side to decode the current picture based on the picture width, the picture height, and the matching window of the current picture and a reference picture for the current picture. 8. Method according to any one of paragraphs. 1-5, further comprising displaying on a display of the electronic device an image generated using the current image. 9. A coding method implemented by a video encoder, containing: generating, by the video encoder, a first picture parameter set (PPS) and a second picture parameter set, each of which refers to the same sequence parameter set (SPS), wherein each of the first picture parameter set and the second picture parameter set comprises pic_width_in_luma_samples, pic_height_in_luma_samples, and match window parameters, wherein pic_width_in_luma_samples indicates the width of each decoded image referring to the corresponding set of image parameters, in units of luma samples, and pic_height_in_luma_samples indicates the height of each decoded image referring to the corresponding set of image parameters, in units of luma samples; wherein, when the first set of image parameters and the second set of image parameters have the same values of pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, the first set of image parameters and the second set of image parameters have the same values of the correspondence window parameters; encoding, by a video encoder, the first set of image parameters and the second set of image parameters into a video bitstream; And storing a video bitstream by the video encoder for transmission to the video decoder. 10. The method of claim 9, wherein the match window comprises a match window left offset, a match window right offset, a match window offset up, and a match window offset down. 11. The method of claim 9 or 10, further comprising transmitting a video bitstream comprising a first set of image parameters and a second set of image parameters to a video decoder. 12. A decoding device comprising: a receiver configured to receive an encoded video bit stream; a memory connected to the receiver that stores instructions; And a processor coupled to the memory, wherein the processor is configured to execute instructions to initiate a decoding device to: receiving a first picture parameter set (PPS) and a second picture parameter set, each of which refers to the same sequence parameter set (SPS), wherein each of the first picture parameter set and the second picture parameter set comprises pic_width_in_luma_samples, pic_height_in_luma_samples, and window parameters matches, wherein pic_width_in_luma_samples indicates the width of each decoded image referring to the corresponding set of image parameters, in units of luma samples, and pic_height_in_luma_samples indicates the height of each decoded image referring to the corresponding set of image parameters, in units of luma samples; wherein, when the first set of image parameters and the second set of image parameters have the same values of pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, the first set of image parameters and the second set of image parameters have the same values of the correspondence window parameters; And applying a match window to the current image corresponding to the first set of image parameters or the second set of image parameters. 13. The decoding apparatus of claim 12, wherein the match window parameters comprise a match window left offset, a match window right offset, a match window up offset, and a match window down offset. 14. The decoding apparatus of claim 12 or 13, wherein the processor is further configured to execute instructions for causing the decoding apparatus to decode a current image corresponding to the first set of image parameters or the second set of image parameters using inter-frame prediction after applying a matching window, wherein the inter-frame the prediction is based on resampling the reference image. 15. The decoding device according to claim 13 or 14, further comprising a display configured to display an image generated based on the current image. 16. An encoding device containing: memory containing instructions; a processor coupled to a memory, the processor configured to execute instructions to initiate an encoder for: generating a first set of image parameters and a second set of image parameters, each of which refers to the same set of sequence parameters, wherein each of the first set of image parameters and the second set of image parameters contains pic_width_in_luma_samples, pic_height_in_luma_samples, and match window parameters, wherein pic_width_in_luma_samples indicates the width of each decoded image referring to the corresponding set of image parameters, in units of luma samples, and pic_height_in_luma_samples indicates the height of each decoded image referring to the corresponding set of image parameters, in units of luma samples; wherein, when the first set of image parameters and the second set of image parameters have the same values of pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, the first set of image parameters and the second set of image parameters have the same values of the correspondence window parameters; And encoding the first set of image parameters and the second set of image parameters into a video bitstream; And storing the video bitstream for transmission to the video decoder; wherein the transmitter is coupled to the processor, wherein the transmitter is configured to transmit a video bitstream comprising a first set of image parameters and a second set of image parameters to the video decoder. 17. The encoding apparatus of claim 16, wherein the match window comprises a match window left offset, a match window right offset, a match window offset up, and a match window offset down. 18. Encoding/decoding equipment containing: a receiver configured to receive an image for encoding or to receive a bit stream for decoding; a transmitter coupled to the receiver, wherein the transmitter is configured to transmit a bit stream to a decoder or to transmit a decoded image to a display; a memory connected to at least one of the receiver or transmitter, the memory being configured to store instructions; and a processor coupled to the memory, wherein the processor is configured to execute instructions stored in the memory to implement the method of any one of claims. 1-8 and any of paragraphs. 9-11. 19. Encoding/decoding equipment according to claim 18, further comprising a display configured to display an image. 20. System for encoding/decoding, containing: encoder; And a decoder coupled to the encoder, wherein the decoder includes a decoding device according to any one of claims. 12-15, and the encoder includes an encoding device according to any one of claims. 16-17. 21. Encoding/decoding means containing: receiving means configured to receive an image for encoding or to receive a bit stream for decoding; transmission means connected to the reception means, wherein the transmission means is configured to transmit a bit stream to the decoding means or transmit a decoded image to the display means; storage means connected to at least one of the receiving means or the transmitting means, wherein the storage means is configured to store instructions; And processing means connected to the storage means, wherein the processing means is configured to execute instructions stored in the storage means to implement the method according to any one of claims. 1-8 and any of paragraphs. 9-11.

Images