WO2025139841A1

WO2025139841A1 - Methods and apparatus of enabling resolution change in loop filter stage for video coding

Info

Publication number: WO2025139841A1
Application number: PCT/CN2024/139042
Authority: WO
Inventors: Guan-hao CHEN; Shih-Chun Chiu; Tzu-Der Chuang; Ching-Yeh Chen; Chih-Wei Hsu; Yi-Wen Chen; Yu-Wen Huang
Original assignee: MediaTek Inc
Current assignee: MediaTek Inc
Priority date: 2023-12-29
Filing date: 2024-12-13
Publication date: 2025-07-03
Anticipated expiration: 2026-06-29
Also published as: TW202527543A

Abstract

A method and apparatus for video coding with in-loop filter at different resolutions form the reconstructed picture are disclosed. According to one method, reconstructed data associated with a current block or first filtered data of the reconstructed data is received. One or more second filters are applied to the reconstructed data on the first filtered data to generate second filtered data, wherein at least one of said one or more second filters is performed with picture rescaling to change pre-filtering resolution to a different resolution. The second filtered data is provided.

Description

METHODS AND APPARATUS OF ENABLING RESOLUTION CHANGE IN LOOP FILTER STAGE FOR VIDEO CODING

CROSS REFERENCE TO RELATED APPLICATIONS

The present invention is a non-Provisional Application of and claims priority to U.S. Provisional Patent Application No. 63/615,816, filed on December 29, 2023. The U.S. Provisional Patent Application is hereby incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to video coding system. In particular, the present invention relates to changing picture resolution for in-loop filter in a video coding system.
BACKGROUND AND RELATED ART

Versatile video coding (VVC) is the latest international video coding standard developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) . The standard has been published as an ISO standard: ISO/IEC 23090-3: 2021, Information technology -Coded representation of immersive media -Part 3: Versatile video coding, published Feb. 2021. VVC is developed based on its predecessor HEVC (High Efficiency Video Coding) by adding more coding tools to improve coding efficiency and also to handle various types of video sources including 3-dimensional (3D) video signals.

Fig. 1A illustrates an exemplary adaptive Inter/Intra video encoding system incorporating loop processing. For Inter/Intra Prediction 110, the prediction data is derived based on previously coded video data in the current picture (i.e., Intra prediction) or previous reconstructed reference picture (s) . The prediction data is subtracted from the input data using Adder 112 to form prediction errors, also called residues. The prediction errors are then processed by Transform (T) 114 followed by Quantization (Q) 116. The transformed and quantized residues are then coded by Entropy Encoding 120 to be included in a video bitstream corresponding to the compressed video data. The bitstream associated with the residues is then packed with side information such as motion and coding modes associated with Inter/Intra prediction, and other information such as parameters associated with loop filters applied to underlying video area. The encoder also needs reconstructed data to derive the inter/intra prediction. Accordingly, the transformed and quantized residues are processed by Inverse Quantization (IQ) 126 and Inverse Transformation (IT) 124 to reconstruct the residues. The residues are then added back to prediction data at Reconstruction (REC) 128 to form reconstructed video data. The reconstructed video data may be stored in Frame Buffer 140 and used for prediction of other frames.

As shown in Fig. 1A, incoming video data undergoes a series of processing in the encoding system. The reconstructed video data from REC 128 may be subject to various impairments due to lossy processing (e.g. quantization) . Accordingly, in-loop filter processing is often applied to the reconstructed video data before the reconstructed video data are stored in the Frame Buffer 140 in order to improve video quality. For example, deblocking filter (DF) 130, Sample Adaptive Offset (SAO) 132 and Adaptive Loop Filter (ALF) 134 may be used. The loop filter information may need to be incorporated in the bitstream so that a decoder can properly recover the required information.

The decoder, as shown in Fig. 1B, can use similar or portion of the reconstruction blocks as the encoder except for Transform 114 and Quantization 116 since the decoder only needs Inverse Quantization 126 and Inverse Transform 124. Instead of Entropy Encoding 120, the decoder uses an Entropy Decoding 160 to decode the video bitstream into quantized transform coefficients and needed coding information (e.g. in-loop filter information, Intra prediction information and Inter prediction information) . The Inter/Intra prediction 150 at the decoder side does not need to perform the Intra mode search nor motion estimation. Instead, the decoder only needs to generate Inter/Intra prediction according to Inter/Intra prediction information received from the encoder. The decoder also uses Frame Buffer to store reconstructed pictures for Inter/Intra prediction. The Inter/Intra prediction data derived by using Inter/Intra Prediction is added to the reconstructed residues from IT 124 using Adder 112. The combined data is provided to REC 128 to form reconstructed pictures.

In VVC, images in the DF 130, SAO 132, and ALF 134 modules as shown in Fig. 1A and Fig. 1B all use the same resolution as those in the REC 128 module. In VVC, one of the target applications is real-time communication, such as video conference and live video. VVC can adapt to dynamic network environment by adaptively adjust picture resolution. Therefore, the encoder can adapt to the worse network condition by encoding low-resolution frames, or transmit full-resolution video when network condition gets better. The above mentioned applications can benefit from supporting spatial resolution changes within a video sequence under unstable network conditions, namely reference picture resampling (RPR) .

In the present invention, methods and apparatus to use adaptive picture resolution in the in-loop filter process are disclosed.
BRIEF SUMMARY OF THE INVENTION

In one embodiment, said one or more second filters comprise DF (Deblocking filter) , SAO (Sample Adaptive Offset) , ALF (Adaptive Loop Filter) , or a combination thereof.

In one embodiment, the picture rescaling corresponds to up-sampling or down-sampling.

In one embodiment, said at least one of said one or more second filters is determined and an on/off control syntax is signalled or parsed to indicate whether said at least one of said one or more second filters is performed with the picture rescaling to change the pre-filtering resolution. In one embodiment, the on/off control syntax is signalled or parsed in a slice level, picture level, sequence level, CTU-level, block level, or a combination thereof.

In one embodiment, an index is signalled or parsed in a slice level, picture level, sequence level, CTU-level, block level, or a combination thereof to indicate whether said at least one of said one or more second filters is rescaled.

In one embodiment, for at least one of said one or more second filters, a number of CTUs being rescaled is changed if CTU size is kept the same as before the picture rescaling. In another embodiment, for at least one of said one or more second filters, CTU size is rescaling to kept a number of CTUs the same as before the picture rescaling. In yet another embodiment, for at least one of said one or more second filters, CTU size is unchanged and a number of CTUs being rescaled is re-calculated.

In one embodiment, for inter-coded pictures, if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a resolution different from the pre-filtering resolution, the filtered reconstruction picture is rescaled to a rescaled-filtered reconstruction picture having the pre-filtering resolution, and the filtered reconstruction picture or the rescaled-filtered reconstruction picture is stored in a buffer as one reference picture. In one embodiment, for inter-coded pictures, if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a resolution different from the pre-filtering resolution, the filtered reconstruction picture is stored in a buffer as one reference picture. In another embodiment, for inter-coded pictures, if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a resolution different from the pre-filtering resolution, the filtered reconstruction picture is rescaled to a rescaled-filtered reconstruction picture having the pre-filtering resolution, and both of the filtered reconstruction picture and the rescaled-filtered reconstruction picture are stored in a buffer as reference pictures.

In one embodiment, a picture-level and/or a slice-level control flag to control resolution of a target second filter is signalled or parsed after a second flag indicating whether the target second filter is on or off. In one embodiment, if the second flag indicates that the target second filter is off, the target second filter is not applied and target picture rescaling is not applied before and/or after the target second filter. In one embodiment, the picture-level and/or the slice-level control flag is signalled or parsed only if the second flag indicates that the target second filter is on. In one embodiment, the picture-level and/or the slice-level control flag is separate for different colour components. In one embodiment, if target resolution for the target second filter is allowed to be different, QP value for the target second filter is adaptively changed according to the target resolution.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1A illustrates an exemplary adaptive Inter/Intra video coding system incorporating loop processing.

Fig. 1B illustrates a corresponding decoder for the encoder in Fig. 1A.

Fig. 2 illustrates an example of possible filter stages for applying different resolution according to embodiments of the present invention.

Fig. 3 illustrates an example of possible filter stages at DF, SAO and ALAF for applying different resolution according to embodiments of the present invention.

Fig. 4 illustrates an example of possible filter stages for applying different resolution and storing filtered reconstructed pictures at two different resolutions in the reference picture buffer according to embodiments of the present invention.

Fig. 5 illustrates a flowchart of an exemplary video coding system that enables different resolution in the in-loop filter according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

It will be readily understood that the components of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the systems and methods of the present invention, as represented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. References throughout this specification to “one embodiment, ” “an embodiment, ” or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, etc. In other instances, well-known structures, or operations are not shown or described in detail to avoid obscuring aspects of the invention. The illustrated embodiments of the invention will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of apparatus and methods that are consistent with the invention as claimed herein.

In the present invention, methods and apparatus to enable in-loop filter at different resolutions are disclosed.

Scheme 1: Process In-Loop Filter at Desired Resolution

In one embodiment, the picture resolution can be different from the pre-LF resolution during in-loop filter (LF) processing at any stage. An example of possible location (or stage) of In-Loop Filter processing at desired resolution is shown in Fig. 2, where LR means low resolution and HR mean high resolution, where REC 128 and Inter/Intra Prediction 110 are performed at low resolution while DF 130, SAO 132 and ALF 134 are performed at high resolution.

For example, one picture is coded in smaller or larger resolution using the VVC reference picture rescaling (RPR) functionality at the pre-LF stage. Before processing the LF, we can rescale the picture to the original resolution or rescale to another resolution that is different from the resolution in pre-LF stage. The LF processing then can be applied at the original resolution or at another resolution that is different from the resolution in the pre-LF stage.

In another example, we can process partial stages of LF at the same resolution as pre-LF stage and do the rest of LF stages at desired resolution. In another example, the rescaling process can be applied after the partial stages LF stages.

For example, Fig. 3 shows the case of performing rescaling before ALF so that the processing in different resolutions is at ALF stage only, where only ALF is performed at HR while others (i.e., REC, DF and SAO) are performed at LR..

In one embodiment, the stage of LF to perform rescaling is fixed and an on/off control syntax is signalled at a high-level (e.g. at slice level, picture level and/or sequence level) , CTU-level, or block level to indicate whether the method is applied or not.

In another embodiment, and index is signalled at a high-level (e.g. slice level, picture level, and/or sequence level) , CTU-level, or block level to indicate which stage of LF to perform rescaling or not perform rescaling.

When the picture is rescaled at one of LF stages, the number of CTUs to be processed in LF stage may be changed if the CTU size is kept the same as pre-LF stage. In one embodiment, the CTU size is also rescaled to keep the number of CTU the same as the pre-LF stage.

For example, if we rescale the picture to make the width and height increased to 2 times, the CTU width and height are also rescaled by 2 times.

In another embodiment, the CTU size is unchanged, and the number of CTUs in LF stage should be re-calculated.

When inter-coded pictures are allowed and the above method is applied, the resolution of reconstruction after LF of the coded picture may be different from the pre-LF resolution of the current picture. In one embodiment, the reconstruction after LF of the coded picture is rescaled to the same resolution as pre-LF resolution of the current picture and put into frame buffer as one of the reference picture. In one embodiment, the reconstruction after LF is stored and put into the frame buffer as reference picture. In another embodiment, two reconstructions are stored into the frame buffer if the resolution is changed during LF. One is the after LF reconstruction and the other one is the same resolution as pre-LF which is rescaled from the after LF one. In another example, the reconstruction picture after LF is always rescaled to the high-resolution (maximum resolution, or the original resolution) if needed. The high-resolution picture can also be put in the reference frame buffer. In yet another example, the coded reference picture is rescaled from the high-resolution picture reconstructed picture to the same resolution as pre-LF of current picture.

For example, we choose to process ALF at different resolution. The reconstruction after ALF can be put into frame buffer directly (path 1) . Also, the reconstruction after ALF can be rescaled to the same resolution as pre-LF (path 2) . In above embodiment, we can use path 1, path 2, or both as shown in Fig. 4.

In one embodiment, a flag is signalled to indicate whether an LF is performed at a higher resolution or not, and the flag is signalled after the on/off control of the LF. Specifically, the flag is signalled only when the LF is turned on.

In another embodiment, an LF can be performed at a resolution different from the pre-LF resolution. If the LF is turned off, the reconstructed picture not only bypasses the LF, but also the rescaling process before and/or after the LF.

For example, we choose to process ALF at a different resolution, and the reconstruction after ALF is rescaled to the same resolution as pre-LF (i.e., path 2 in Fig. 4) . If ALF is turned off, not only ALF is bypassed, but also the rescaling process before and after ALF are bypassed.

In the above embodiments, the design can be separate for different luma/chroma components.

For example, a flag indicating whether a luma/chroma LF is performed at a higher resolution or not is signalled after the on/off control of the luma/chroma LF.

For another example, we choose to process ALF at a different resolution, and the reconstruction after ALF is rescaled to the same resolution as pre-LF (path 2 in Fig. 4) . If luma ALF is turned on but chroma ALF and CCALF are turned off, the luma component is processed normally, but for the chroma components, both ALF and the rescaling process before and after ALF are bypassed.

In one embodiment, LF can be processed at different resolutions. The QP value for the LF stage is adaptively changed according to the resolution.

For example, if ALF is processed at the RPR (down-scaled) resolution, the QP value for ALF fixed filter selection follows the signalled QP value; while if ALF is processed at the original resolution, the QP value for ALF fixed filter selection will be adjusted according to the RPR scaling ratio.

Any of the foregoing proposed methods of applying in-loop filter at different resolution can be implemented in encoders and/or decoders. For example, any of the proposed methods can be implemented in one module of an encoders and/or decoder. Alternatively, any of the proposed methods can be implemented as a circuit coupled to one module of the encoders and/or decoder, so as to provide the information needed by the module used in encoders and/or decoder. The proposed methods of applying in-loop filter at different resolution can be implemented in an encoder side or a decoder side. For example, with reference to the encoder and decoder in Fig. 1A and Fig. 1B, any of the proposed methods can be implemented in an Intra/Inter coding module (e.g. Intra Pred. 150/MC 152 in Fig. 1B) in a decoder or an Intra/Inter coding module is an encoder (e.g. Intra Pred. 110/Inter Pred. 112 in Fig. 1A) . However, the decoder or encoder may also use additional processing unit to implement the proposed method. While the Intra Pred. units (e.g. unit 110/112 in Fig. 1A and unit 150/152 in Fig. 1B) are shown as individual processing units, they may correspond to executable software or firmware codes stored on a media, such as hard disk or flash memory, for a CPU (Central Processing Unit) or programmable devices (e.g. DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array)) .

Fig. 5 illustrates a flowchart of an exemplary video coding system that enables different resolution in the in-loop filter according to an embodiment of the present invention. The steps shown in the flowchart may be implemented as program codes executable on one or more processors (e.g., one or more CPUs) at the encoder side. The steps shown in the flowchart may also be implemented based hardware such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to one method, reconstructed data associated with a current block or first filtered data of the reconstructed data is received in step 510. One or more second filters are applied to the reconstructed data on the first filtered data to generate second filtered data in step 520, wherein at least one of said one or more second filters is performed with picture rescaling to change pre-filtering resolution to a different resolution. The second filtered data is provided in step 530.

The flowchart shown is intended to illustrate an example of video coding according to the present invention. A person skilled in the art may modify each step, re-arranges the steps, split a step, or combine steps to practice the present invention without departing from the spirit of the present invention. In the disclosure, specific syntax and semantics have been used to illustrate examples to implement embodiments of the present invention. A skilled person may practice the present invention by substituting the syntax and semantics with equivalent syntax and semantics without departing from the spirit of the present invention.

The above description is presented to enable a person of ordinary skill in the art to practice the present invention as provided in the context of a particular application and its requirement. Various modifications to the described embodiments will be apparent to those with skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed. In the above detailed description, various specific details are illustrated in order to provide a thorough understanding of the present invention. Nevertheless, it will be understood by those skilled in the art that the present invention may be practiced.

Embodiment of the present invention as described above may be implemented in various hardware, software codes, or a combination of both. For example, an embodiment of the present invention can be one or more circuit circuits integrated into a video compression chip or program code integrated into video compression software to perform the processing described herein. An embodiment of the present invention may also be program code to be executed on a Digital Signal Processor (DSP) to perform the processing described herein. The invention may also involve a number of functions to be performed by a computer processor, a digital signal processor, a microprocessor, or field programmable gate array (FPGA) . These processors can be configured to perform particular tasks according to the invention, by executing machine-readable software code or firmware code that defines the particular methods embodied by the invention. The software code or firmware code may be developed in different programming languages and different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, styles and languages of software codes and other means of configuring code to perform the tasks in accordance with the invention will not depart from the spirit and scope of the invention.

The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

A method of video coding, the method comprising:

receiving reconstructed data associated with a current block or first filtered data of the reconstructed data;

applying one or more second filters to the reconstructed data on the first filtered data to generate second filtered data, wherein at least one of said one or more second filters is performed with picture rescaling to change pre-filtering resolution to a different resolution; and

providing the second filtered data.
The method of Claim 1, wherein said one or more second filters comprise DF (Deblocking filter) , SAO (Sample Adaptive Offset) , ALF (Adaptive Loop Filter) , or a combination thereof.
The method of Claim 1, wherein the picture rescaling corresponds to up-sampling or down-sampling.
The method of Claim 1, wherein said at least one of said one or more second filters is determined and an on/off control syntax is signalled or parsed to indicate whether said at least one of said one or more second filters is performed with the picture rescaling to change the pre-filtering resolution.
The method of Claim 4, wherein the on/off control syntax is signalled or parsed in a slice level, picture level, sequence level, CTU-level, block level, or a combination thereof.
The method of Claim 1, wherein an index is signalled or parsed in a slice level, picture level, sequence level, CTU-level, block level, or a combination thereof to indicate whether said at least one of said one or more second filters is rescaled.
The method of Claim 1, wherein for at least one of said one or more second filters, a number of CTUs being rescaled is changed if CTU size is kept the same as before the picture rescaling.
The method of Claim 1, wherein for at least one of said one or more second filters, CTU size is rescaling to kept a number of CTUs the same as before the picture rescaling.
The method of Claim 1, wherein for at least one of said one or more second filters, CTU size is unchanged and a number of CTUs being rescaled is re-calculated.
The method of Claim 1, wherein for inter-coded pictures, if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a resolution different from the pre-filtering resolution, the filtered reconstruction picture is rescaled to a rescaled-filtered reconstruction picture having the pre-filtering resolution, and the filtered reconstruction picture or the rescaled-filtered reconstruction picture is stored in a buffer as one reference picture.
The method of Claim 1, wherein for inter-coded pictures, if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a resolution different from the pre-filtering resolution, the filtered reconstruction picture is stored in a buffer as one reference picture.
The method of Claim 1, wherein for inter-coded pictures, if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a resolution different from the pre-filtering resolution, the filtered reconstruction picture is rescaled to a rescaled-filtered reconstruction picture having the pre-filtering resolution, and both of the filtered reconstruction picture and the rescaled-filtered reconstruction picture are stored in a buffer as reference pictures.
The method of Claim 1, wherein if a filtered reconstruction picture corresponding to after said applying said one or more second filters to a reconstruction picture has a same resolution as the pre-filtering resolution, the filtered reconstruction picture is stored in a buffer as one reference picture.
The method of Claim 1, wherein a picture-level and/or a slice-level control flag to control resolution of a target second filter is signalled or parsed after a second flag indicating whether the target second filter is on or off.
The method of Claim 14, wherein if the second flag indicates that the target second filter is off, the target second filter is not applied and target picture rescaling is not applied before and/or after the target second filter.
The method of Claim 14, wherein the picture-level and/or the slice-level control flag is signalled or parsed only if the second flag indicates that the target second filter is on.
The method of Claim 14, wherein the picture-level and/or the slice-level control flag is separate for different colour components.
The method of Claim 14, wherein if target resolution for the target second filter is allowed to be different, QP value for the target second filter is adaptively changed according to the target resolution.
An apparatus for video coding, the apparatus comprising one or more electronics or processors arranged to:

receive reconstructed data associated with a current block or first filtered data of the reconstructed data;

apply one or more second filters to the reconstructed data on the first filtered data to generate second filtered data, wherein at least one of said one or more second filters is performed with picture rescaling to change pre-filtering resolution to a different resolution; and

provide the second filtered data.