WO2022114669A2 - Image encoding using neural network - Google Patents

Abstract

As a disclosure relating to a method for encoding an image (video) by using a neural network, the present embodiment provides an image encoding method for performing versatile video coding (VVC), wherein a ternary tree is determined among multi-type trees (MTTs) by using a light-weight neural network (LNN) on the basis of an explicit attribute and a derivative attribute extracted in a coding unit (CU) encoding process, in order to reduce computational complexity for intra prediction.

Description

Image encoding using neural networks

The present disclosure relates to encoding of an image (video) using a neural network.

The content described below merely provides background information related to the present invention and does not constitute the prior art.

Since video data has a large amount of data compared to audio data or still image data, it requires a lot of hardware resources including memory to store or transmit itself without compression processing.

Accordingly, in general, when storing or transmitting video data, an encoder is used to compress and store or transmit the video data, and a decoder receives, decompresses, and reproduces the compressed video data. As such video compression technologies, there are H.264/AVC, High Efficiency Video Coding (HEVC), and the like, as well as Versatile Video Coding (VVC), which improves coding efficiency by about 30% or more compared to HEVC.

However, as the size, resolution, and frame rate of an image are gradually increasing, and the amount of data to be encoded is increasing accordingly, a new compression technique with better encoding efficiency and higher image quality improvement than the existing compression techniques is required.

In VVC video (video) encoding, a multi-type tree (MTT) is applied to increase encoding efficiency, and thus a binary tree (BT) and a ternary tree (TT) are used. However, MTT, particularly ternary tree, has a problem in that it significantly increases the computational complexity of the video encoding process.

Therefore, a method for efficiently determining the ternary tree should be considered.

The present disclosure relates to a video encoding method for performing versatile video coding (VVC), in order to reduce computational complexity for intra prediction, weight reduction based on explicit and derivative properties extracted in a coding unit (CU) encoding process An object of the present invention is to provide an image encoding method for determining a ternary tree among multi-type trees (MTT) using a light-weight neural network (LNN).

According to an embodiment of the present disclosure, for a tree structure including multi-type tree (MTT)-based block division, determining the division type and direction of a target CU (Coding Unit) corresponding to any one node of the tree structure A method of encoding an image for performing an encoding test for generating an explicit characteristic and a derivative characteristic for the target CU, wherein the explicit characteristic includes at least some of coding information generated according to an encoding test for the target CU, and the derivative characteristic is the encoding costs and derived based on at least one of a ternary tree (TT) division direction to be tested; and determining whether to skip an encoding test for TT segmentation in a horizontal or vertical direction by inputting the explicit and derived characteristics into one or more deep neural networks.

According to another embodiment of the present disclosure, generating a screening result by performing a screening test based on a structure of a dual tree for the target CU and a structure of the MTT, the method further comprising: When the screening result is true, the step of determining whether to skip the encoding test is performed, and when the screening result is false, the horizontal TT division and the vertical TT division are performed to generate a corresponding encoding cost. An image encoding method is provided.

According to another embodiment of the present disclosure, with respect to a tree structure including multi-type tree (MTT)-based block division, a division type and direction of a target coding unit (CU) corresponding to any one node of the tree structure is determined a cost estimator for performing an encoding test on the target CU to generate encoding costs for each of non-segmentation, quad tree (QT) segmentation, and binary tree (BT) segmentation; a feature extracting unit for generating an explicit feature and a derivative feature for the target CU, the explicit feature includes at least some of coding information generated according to an encoding test for the target CU, the derivative feature is the encoding derived based on at least one of costs and a ternary tree (TT) splitting direction to be tested; and a first TT division unit that determines whether to skip an encoding test for TT division in a horizontal or vertical direction by inputting the explicit characteristics and derivative characteristics into one or more deep neural networks. to provide.

According to another embodiment of the present disclosure, a screening test unit for generating a screening result by performing a screening test on the target CU based on a structure of a dual tree (dual tree) and the structure of the MTT; and a second TT division unit, wherein when the screening result is true, the first TT division unit determines whether to skip the encoding test for the TT division in the horizontal or vertical direction, and the screening result is false. In this case, the second TT division unit performs TT division in the horizontal and vertical directions to generate corresponding encoding costs, respectively.

As described above, according to this embodiment, a ternary tree is determined during MTT (Multi-Type Tree) using a lightweight neural network based on explicit and derived characteristics extracted in a coding unit (CU) encoding process. By providing the video encoding method, it is possible to reduce the depth of the MTT structure for intra prediction and thus effectively reduce the computational complexity of the video encoding method for performing VVC.

1 is an exemplary block diagram of an image encoding apparatus that can implement techniques of the present disclosure.

2 is a diagram for explaining a method of dividing a block using a QTBTTT structure.

3 is a diagram illustrating a plurality of intra prediction modes including wide-angle intra prediction modes.

4 is an exemplary diagram of a neighboring block of the current block.

5 is a schematic flowchart of rate-distortion optimization in intra prediction among video encoding methods.

6 is a schematic flowchart of rate-distortion optimization in intra prediction according to an embodiment of the present disclosure.

7 is a schematic illustration of a method for determining a ternary tree according to an embodiment of the present disclosure.

8 is a schematic block diagram of an image encoding apparatus for determining a terrestrial tree according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in the description of the present embodiments, if it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present embodiments, the detailed description thereof will be omitted.

1 is an exemplary block diagram of an image encoding apparatus that can implement techniques of the present disclosure. Hereinafter, an image encoding apparatus for performing versatile video coding (VVC) and sub-configurations of the apparatus will be described with reference to FIG. 1 .

The image encoding apparatus includes a picture division unit 110 , a prediction unit 120 , a subtractor 130 , a transform unit 140 , a quantization unit 145 , a reordering unit 150 , an entropy encoding unit 155 , and an inverse quantization unit. 160 , an inverse transform unit 165 , an adder 170 , a loop filter unit 180 , and a memory 190 may be included.

Each component of the image encoding apparatus may be implemented as hardware or software, or a combination of hardware and software. In addition, the function of each component may be implemented as software and the microprocessor may be implemented to execute the function of software corresponding to each component.

One image (video) is composed of one or more sequences including a plurality of pictures. Each picture is divided into a plurality of regions, and encoding is performed for each region. For example, one picture is divided into one or more tiles and/or slices. Here, one or more tiles may be defined as a tile group. Each tile or/slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more CUs (Coding Units) by a tree structure. Information applied to each CU is encoded as a syntax of the CU, and information commonly applied to CUs included in one CTU is encoded as a syntax of the CTU. In addition, information commonly applied to all blocks in one slice is encoded as a syntax of a slice header, and information applied to all blocks constituting one or more pictures is a picture parameter set (PPS) or a picture. encoded in the header. Furthermore, information commonly referenced by a plurality of pictures is encoded in a sequence parameter set (SPS). In addition, information commonly referred to by one or more SPSs is encoded in a video parameter set (VPS). Also, information commonly applied to one tile or tile group may be encoded as a syntax of a tile or tile group header. Syntax included in the SPS, PPS, slice header, tile or tile group header may be referred to as high-level syntax.

The picture divider 110 determines the size of a coding tree unit (CTU). Information on the size of the CTU (CTU size) is encoded as a syntax of the SPS or PPS and transmitted to the video decoding apparatus.

The picture divider 110 divides each picture constituting an image into a plurality of coding tree units (CTUs) having a predetermined size, and then repeatedly divides the CTUs using a tree structure. (recursively) divide. A leaf node in the tree structure becomes a coding unit (CU), which is a basic unit of encoding.

As a tree structure, a quadtree (QT) in which a parent node (or parent node) is divided into four child nodes (or child nodes) of the same size, or a binary tree (BinaryTree) in which a parent node is divided into two child nodes , BT), or a ternary tree (TT) in which a parent node is divided into three child nodes in a 1:2:1 ratio, or a structure in which two or more of these QT structures, BT structures, and TT structures are mixed have. For example, a QuadTree plus BinaryTree (QTBT) structure may be used, or a QuadTree plus BinaryTree TernaryTree (QTBTTT) structure may be used. Here, BTTT may be combined to be referred to as a Multiple-Type Tree (MTT).

2 is a diagram for explaining a method of dividing a block using a QTBTTT structure.

As shown in FIG. 2 , the CTU may be first divided into a QT structure. The quadtree splitting may be repeated until the size of a splitting block reaches the minimum block size of a leaf node (MinQTSize) allowed in QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is divided into four nodes of a lower layer is encoded by the entropy encoder 155 and signaled to the image decoding apparatus. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further divided into any one or more of the BT structure or the TT structure. A plurality of division directions may exist in the BT structure and/or the TT structure. For example, there may be two directions in which the block of the corresponding node is divided horizontally and vertically. As shown in FIG. 2 , when MTT splitting starts, a second flag (mtt_split_flag) indicating whether or not nodes are split, and a flag indicating additionally splitting direction (vertical or horizontal) if split and/or splitting type (Binary) or Ternary) is encoded by the entropy encoder 155 and signaled to the video decoding apparatus.

Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split is encoded it might be When the CU split flag (split_cu_flag) value indicates that it is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a coding unit (CU), which is a basic unit of coding. When the CU split flag (split_cu_flag) value indicates to be split, the image encoding apparatus starts encoding from the first flag in the above-described manner.

When QTBT is used as another example of the tree structure, there are two types of splitting the block of the node into two blocks of the same size horizontally (ie, symmetric horizontal splitting) and vertically (ie, symmetric vertical splitting). branches may exist. A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating a split type are encoded by the entropy encoder 155 and transmitted to the image decoding apparatus. On the other hand, a type for dividing the block of the corresponding node into two blocks having an asymmetric shape may further exist. The asymmetric form may include a form in which the block of the corresponding node is divided into two rectangular blocks having a size ratio of 1:3, or a form in which the block of the corresponding node is divided in a diagonal direction.

A CU may have various sizes depending on the QTBT or QTBTTT split from the CTU. Hereinafter, a block corresponding to a CU to be encoded or decoded (ie, a leaf node of QTBTTT) is referred to as a 'current block'. According to the adoption of QTBTTT partitioning, the shape of the current block may be not only a square but also a rectangle.

The prediction unit 120 generates a prediction block by predicting the current block. The prediction unit 120 includes an intra prediction unit 122 and an inter prediction unit 124 .

In general, each of the current blocks in a picture may be predictively coded. In general, prediction of the current block is performed using an intra prediction technique (using data from the picture containing the current block) or inter prediction technique (using data from a picture coded before the picture containing the current block). can be performed. Inter prediction includes both uni-prediction and bi-prediction.

The intra prediction unit 122 predicts pixels in the current block by using pixels (reference pixels) located around the current block in the current picture including the current block. A plurality of intra prediction modes exist according to a prediction direction. For example, as shown in FIG. 3A , the plurality of intra prediction modes may include two non-directional modes including a planar mode and a DC mode and 65 directional modes. According to each prediction mode, the neighboring pixels to be used and the calculation expression are defined differently.

For efficient directional prediction of a rectangular-shaped current block, directional modes (Nos. 67 to 80 and No. -1 to No. -14 intra prediction modes) indicated by dotted arrows in FIG. 3B may be additionally used. These may be referred to as “wide angle intra-prediction modes”. Arrows in FIG. 3B indicate corresponding reference samples used for prediction, not prediction directions. The prediction direction is opposite to the direction indicated by the arrow. The wide-angle intra prediction modes are modes in which a specific directional mode is predicted in the opposite direction without additional bit transmission when the current block is rectangular. In this case, among the wide-angle intra prediction modes, some wide-angle intra prediction modes available for the current block may be determined by the ratio of the width to the height of the rectangular current block. For example, the wide-angle intra prediction modes having an angle smaller than 45 degrees (intra prediction modes 67 to 80) are available when the current block has a rectangular shape with a height smaller than the width, and a wide angle having an angle greater than -135 degrees. The intra prediction modes (intra prediction modes -1 to -14) are available when the current block has a rectangular shape with a width greater than a height.

The intra prediction unit 122 may determine an intra prediction mode to be used for encoding the current block. In some examples, the intra prediction unit 122 may encode the current block using several intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit 122 calculates rate distortion values using rate-distortion (RD) analysis for several tested intra prediction modes, and has the best rate distortion characteristic among the tested modes. It is also possible to select an intra prediction mode having .

The intra prediction unit 122 selects one intra prediction mode from among a plurality of intra prediction modes, and predicts the current block by using a neighboring pixel (reference pixel) determined according to the selected intra prediction mode and an equation. Information on the selected intra prediction mode is encoded by the entropy encoder 155 and transmitted to an image decoding apparatus.

The inter prediction unit 124 generates a prediction block for the current block by using a motion compensation process. The inter prediction unit 124 searches for a block most similar to the current block in the reference picture encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to displacement between the current block in the current picture and the prediction block in the reference picture is generated. In general, motion estimation is performed for a luma component, and a motion vector calculated based on the luma component is used for both the luma component and the chroma component. Motion information including information on a reference picture and information on a motion vector used to predict the current block is encoded by the entropy encoder 155 and transmitted to the image decoding apparatus.

The inter prediction unit 124 may perform interpolation on a reference picture or reference block in order to increase prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When the process of searching for a block most similar to the current block is performed with respect to the interpolated reference picture, the motion vector can be expressed up to the precision of the decimal unit rather than the precision of the integer sample unit. The precision or resolution of the motion vector may be set differently for each unit of a target region to be encoded, for example, a slice, a tile, a CTU, or a CU. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, information on motion vector resolution applied to each CU is signaled. The information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

Meanwhile, the inter prediction unit 124 may perform inter prediction using bi-prediction. In the case of bi-directional prediction, two reference pictures and two motion vectors indicating the position of a block most similar to the current block in each reference picture are used. The inter prediction unit 124 selects a first reference picture and a second reference picture from the reference picture list 0 (RefPicList0) and the reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block in each reference picture. A first reference block and a second reference block are generated. Then, the first reference block and the second reference block are averaged or weighted to generate a prediction block for the current block. In addition, motion information including information on two reference pictures and information on two motion vectors used to predict the current block is transmitted to the encoder 150 . Here, reference picture list 0 consists of pictures before the current picture in display order among the restored pictures, and reference picture list 1 consists of pictures after the current picture in display order among the restored pictures. have. However, the present invention is not necessarily limited thereto, and in display order, the restored pictures after the current picture may be further included in the reference picture list 0, and conversely, the restored pictures before the current picture are additionally added to the reference picture list 1. may be included.

Various methods may be used to minimize the amount of bits required to encode motion information.

For example, when the reference picture and motion vector of the current block are the same as the reference picture and motion vector of the neighboring block, the motion information of the current block may be transmitted to the image decoding apparatus by encoding information for identifying the neighboring block. This method is called 'merge mode'.

In the merge mode, the inter prediction unit 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from neighboring blocks of the current block.

As the neighboring blocks for deriving the merge candidate, as shown in FIG. 4 , the left block (L), the upper block (A), the upper right block (AR), and the lower left block (BL) adjacent to the current block in the current picture. ), all or part of the upper left block AL may be used. In addition, a block located in a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block in the reference picture or blocks adjacent to the co-located block may be further used as merge candidates.

The inter prediction unit 124 constructs a merge list including a predetermined number of merge candidates by using these neighboring blocks. A merge candidate to be used as motion information of the current block is selected from among the merge candidates included in the merge list, and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the encoder 150 and transmitted to the image decoding apparatus.

Another method for encoding motion information is AMVP (Advanced Motion Vector Prediction) mode.

In the AMVP mode, the inter prediction unit 124 derives motion vector prediction candidates for the motion vector of the current block using neighboring blocks of the current block. As the neighboring blocks used to derive the prediction motion vector candidates, the left block (L), the upper block (A), the upper right block (AR), the lower left block ( BL), all or part of the upper left block AL may be used. In addition, a block located in a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture in which the current block is located is used as a neighboring block used to derive prediction motion vector candidates. may be For example, a block co-located with the current block in the reference picture or blocks adjacent to the co-located block may be used.

The inter prediction unit 124 derives prediction motion vector candidates by using the motion vectors of the neighboring blocks, and determines a predicted motion vector with respect to the motion vector of the current block by using the prediction motion vector candidates. Then, a differential motion vector is calculated by subtracting the predicted motion vector from the motion vector of the current block.

The prediction motion vector may be obtained by applying a predefined function (eg, a median value, an average value operation, etc.) to the prediction motion vector candidates. In this case, the image decoding apparatus also knows the predefined function. Also, since the neighboring block used to derive the prediction motion vector candidate is a block that has already been encoded and decoded, the video decoding apparatus already knows the motion vector of the neighboring block. Therefore, the image encoding apparatus does not need to encode information for identifying the prediction motion vector candidate. Accordingly, in this case, information on a differential motion vector and information on a reference picture used to predict a current block are encoded.

Meanwhile, the prediction motion vector may be determined by selecting any one of the prediction motion vector candidates. In this case, information for identifying the selected prediction motion vector candidate is additionally encoded together with information on the differential motion vector and information on the reference picture used to predict the current block.

The subtractor 130 generates a residual block by subtracting the prediction block generated by the intra prediction unit 122 or the inter prediction unit 124 from the current block.

The transform unit 140 transforms the residual signal in the residual block having pixel values in the spatial domain into transform coefficients in the frequency domain. The transform unit 140 may transform the residual signals in the residual block by using the entire size of the residual block as a transform unit, or divide the residual block into a plurality of sub-blocks and use the sub-blocks as transform units to perform transformation. You may. Alternatively, the residual signals may be transformed by dividing the sub-block into two sub-blocks, which are a transform region and a non-transform region, and use only the transform region sub-block as a transform unit. Here, the transform region subblock may be one of two rectangular blocks having a size ratio of 1:1 based on the horizontal axis (or vertical axis). In this case, the flag (cu_sbt_flag) indicating that only the subblock has been transformed, the vertical/horizontal information (cu_sbt_horizontal_flag), and/or the position information (cu_sbt_pos_flag) are encoded by the entropy encoder 155 and signaled to the video decoding apparatus. do. Also, the size of the transform region subblock may have a size ratio of 1:3 based on the horizontal axis (or vertical axis). Signaled to the decoding device.

Meanwhile, the transform unit 140 may individually transform the residual block in a horizontal direction and a vertical direction. For transformation, various types of transformation functions or transformation matrices may be used. For example, a pair of transform functions for horizontal transformation and vertical transformation may be selected (Multiple Transform Selection: MTS). The transform unit 140 may select one transform function pair having the best transform efficiency by using the MTS and transform the residual block in horizontal and vertical directions, respectively. The information (mts_idx) on the transform function pair selected according to the MTS is encoded by the entropy encoder 155 and signaled to the image decoding apparatus.

The quantization unit 145 quantizes the transform coefficients output from the transform unit 140 using a quantization parameter, and outputs the quantized transform coefficients to the entropy encoding unit 155 . The quantization unit 145 may directly quantize a related residual block for a certain block or frame without transformation. The quantization unit 145 may apply different quantization coefficients (scaling values) according to positions of the transform coefficients in the transform block. A quantization matrix applied to two-dimensionally arranged quantized transform coefficients may be encoded and signaled to an image decoding apparatus.

The rearrangement unit 150 may rearrange the coefficient values on the quantized residual values.

The reordering unit 150 may change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit 150 may output a one-dimensional coefficient sequence by scanning from DC coefficients to coefficients in a high frequency region using a zig-zag scan or a diagonal scan. . A vertical scan for scanning a two-dimensional coefficient array in a column direction and a horizontal scan for scanning a two-dimensional block shape coefficient in a row direction may be used instead of the zig-zag scan according to the size of the transform unit and the intra prediction mode. That is, a scanning method to be used among a zig-zag scan, a diagonal scan, a vertical scan, and a horizontal scan may be determined according to the size of the transform unit and the intra prediction mode.

The entropy encoding unit 155 uses various encoding methods such as Context-based Adaptive Binary Arithmetic Code (CABAC) and Exponential Golomb to convert the one-dimensional quantized transform coefficients output from the reordering unit 150 . A bitstream is created by encoding the sequence.

In addition, the entropy encoding unit 155 encodes information such as CTU size, CU split flag, QT split flag, MTT split type, and MTT split direction related to block splitting, so that the video decoding apparatus divides the block in the same way as the video encoding apparatus. to be able to divide. In addition, the entropy encoder 155 encodes information on a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and intra prediction information (ie, intra prediction) according to the prediction type. mode) or inter prediction information (in the case of the merge mode, the merge index, in the case of the AMVP mode, the reference picture index and the information on the differential motion vector) is encoded. Also, the entropy encoder 155 encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

The inverse quantization unit 160 inverse quantizes the quantized transform coefficients output from the quantization unit 145 to generate transform coefficients. The inverse transform unit 165 reconstructs a residual block by transforming the transform coefficients output from the inverse quantization unit 160 from the frequency domain to the spatial domain.

The addition unit 170 restores the current block by adding the reconstructed residual block to the prediction block generated by the prediction unit 120 . Pixels in the reconstructed current block are used as reference pixels when intra-predicting the next block.

The loop filter unit 180 reconstructs pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. generated due to block-based prediction and transformation/quantization. filter on them. The loop filter unit 180 may include all or part of a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186 as an in-loop filter. have.

The deblocking filter 182 filters the boundary between the reconstructed blocks in order to remove a blocking artifact caused by block-by-block encoding/decoding, and the SAO filter 184 and the ALF 186 deblocking filtering Additional filtering is performed on the captured image. The SAO filter 184 and the ALF 186 are filters used to compensate for a difference between a reconstructed pixel and an original pixel caused by lossy coding. The SAO filter 184 improves encoding efficiency as well as subjective image quality by applying an offset in units of CTUs. On the other hand, the ALF 186 performs block-by-block filtering, and the distortion is compensated by applying different filters by classifying the edge of the corresponding block and the degree of change. Information on filter coefficients to be used in the ALF 186 may be encoded and signaled to an image decoding apparatus.

The restored block filtered through the deblocking filter 182 , the SAO filter 184 and the ALF 186 is stored in the memory 190 . When all blocks in one picture are reconstructed, the reconstructed picture may be used as a reference picture for inter prediction of blocks in a picture to be encoded later.

This embodiment relates to encoding of an image (video) using a neural network. More specifically, in the video encoding method for performing versatile video coding (VVC), in order to reduce the computational complexity for intra prediction, based on the explicit and derived characteristics extracted in the coding unit (CU) encoding process, An image encoding method for determining a ternary tree among multi-type trees (MTT) using a light-weight neural network (LNN) is provided.

Hereinafter, in the description according to the present embodiment, with respect to a tree structure including a multi-type tree (MTT)-based block division, a block corresponding to one node of the tree structure is indicated as a CU. Also, a CU corresponds to a current block to be encoded.

5 is a schematic flowchart of rate-distortion optimization in intra prediction among video encoding methods.

An apparatus for encoding an image performing VVC may determine a split type and direction of a CU based on rate-distortion (RD) optimization and split the CU into blocks of an MTT structure.

The image encoding apparatus performs intra prediction on the CU (S500). The image encoding apparatus calculates the cost value J _cur for the current CU according to rate-distortion by performing intra prediction and transformation on the CU. Here, the process of calculating J _cur may be subdivided as follows based on a plurality of coding tools.

The image encoding apparatus performs a rate-distortion test on 67 intra prediction modes as shown in FIG. 3A ( S520 ), and uses up to 3 neighboring reference lines for intra prediction, Multiple Reference Line (MRL) to perform a rate-distortion test (S522). In addition, the video encoding apparatus, for example, performs a rate-distortion test by applying an Intra Subblock Partition (ISP) up to five division types (S524), and performs a rate-distortion test by applying a Transform Selection (TS). and (S526), for example, by applying MTS up to five transform pairs, a rate-distortion test is performed (S528). The image encoding apparatus may perform appropriate intra prediction for the CU based on the detailed procedures S520 to S528 and calculate the cost value J _cur accordingly.

On the other hand, when additional division of the CU is possible, the image encoding apparatus may perform a test based on MTT division.

The video encoding apparatus is QT partitioning, BT_H (Binary Tree Horizontal) partitioning, BT_H (Binary Tree Vertical, vertical binary tree) partitioning, TT_H (Ternary Tree Horizontal, horizontal ternary tree) partitioning, and TT_V (Ternary Tree Vertical, TT_V) partitioning. A rate-distortion test is performed by sequentially applying the vertical ternary tree) partitioning (S502 to S510). The video encoding apparatus may calculate costs J _QT , J _{BT_H} , J _{BT_V} , J _{TT_H} and J _{TT_V} according to the MTT division application of QT, BT_H, BT_V, TT_H, and TT_V.

The image encoding apparatus determines an intra prediction mode and a partition structure for a CU based on the calculated costs (S512).

Hereinafter, an image encoding method for determining a ternary tree according to the present embodiment based on rate-distortion optimization in order to reduce the computational complexity of intra prediction based on MTT segmentation will be described with reference to FIGS. 6 and 7 . do.

The video encoding apparatus according to the present disclosure additionally performs a new process (S600 to S616 in the example of FIG. 6 ) using LNN in addition to the rate-distortion optimization method as illustrated in FIG. 5 .

6 is a schematic flowchart of rate-distortion optimization in intra prediction according to an embodiment of the present disclosure.

7 is a schematic illustration of a method for determining a ternary tree according to an embodiment of the present disclosure.

As illustrated in FIGS. 6 and 7 , the process of determining the ternary tree includes feature extraction (S600 and S602), a screening test (S604), and an early TT division (S606 to S616) according to the result, or an existing normal TT segmentation (S508 and S510).

The video encoding apparatus extracts a CU segmentation characteristic for use as an input of the LNN in the intra prediction, QT segmentation, and BT segmentation steps (S500 to S506) of the CU (S600 and S602). These CU partitioning characteristics are, (i) coding information according to intra prediction, (ii) parameters according to QT (Quad Tree) partitioning and MTT (Multi-Type Tree) partitioning, (iii) CU, QT partitioning and BT (Binary Tree) partitioning. ) can be extracted based on the RD cost according to the division. As shown in Table 1, the CU segmentation feature may include an explicit feature (EF) and a derived feature (DF).

Here, a Coded Block Flag (CBF) is generated based on a flag indicating whether a transform tree structure for a CU exists, and an MTD is generated based on the depth of the MTT (ie, BT/TT) division. BSR (Block Shape Ratio) is a ratio of the block shape according to the direction of TT division, and is defined as lh/(lw+lh) in the case of TT_H division, and is defined as lw/(lw+lh) in the case of TT_V division, lw and lh respectively indicate the width and height of the luma block. BTD (BT Direction) is a Boolean value indicating whether the optimal direction for BT division in terms of RD cost coincides with the direction of TT division to be tested. BTS (BTs' Superiority) is a characteristic for indicating the superiority of BT division in terms of RD cost, and is set to 0.5 when one of J _{BT_H} or J _{BT_V} is smaller than the other (J _cur and J _CU ), J _{BT_H} and It is set to 1 if both J _{BT_V} are less than the other cases (J _cur and J _CU ).

Meanwhile, all properties shown in Table 1 are normalized to values ranging from 0 to 1.

In order to reduce the complexity of the LNN, among the properties shown in Table 1, only some properties helpful in improving the accuracy of the LNN may be used. Also, strongly correlated items of the illustrated properties can be combined to reduce overhead without sacrificing the accuracy of the LNN. In order to understand the importance of the features shown in Table 1, a Pearson Correlation Coefficient (PCC) indicating a linear correlation between the features and the binary class may be calculated. Here, the binary class is defined as (x _i , y _i ) for n feature vectors x _i (1≤i≤n), and y _i is included in {0,1}. (y _i =1) indicates that TT is the optimal choice for CU in terms of RD cost. On the other hand, (y _i =0) indicates that it is not an optimal choice for the CU. The Pearson correlation coefficient between the features shown in Table 1 and the binary class can be calculated by encoding the test image using an LNN based on the test image and the features shown in Table 1, which can be exemplified as shown in Table 2 have.

As shown in Table 2, the four characteristics, IPM, QP, MRL and MTS, show a low correlation with the binary class from -0.07 to 0.02. In addition, when the PCC value between QTD and MTD is obtained, since there is a strong correlation as -0,65, only one of the two can be used. This strong correlation also exists between ISPs and CBFs. Therefore, in the present disclosure, among the characteristics shown in Table 1, as shown in Table 2, in the order of the significant correlation with the binary class, five, BSR, BTD, BTS, MTD, and CBF, are selected as final characteristics. It is used as input for LNN. Among them, MTD and CBF are values that can be directly calculated from a coding tool and a coding parameter after intra prediction for the current CU, and are thus denoted by an explicit characteristic x _EF . Since the remaining three characteristics are values that can be calculated based on the block shape and RD cost after BT division is performed, they are expressed as derived characteristics x _DF .

Meanwhile, the derivative feature x _DF may be generated in various situations. First, it can be assumed that the derivative characteristic according to the present disclosure is applied when additional MTT splitting for a CU is allowed. This assumption includes the special case where QT partitioning is not allowed for a CU, but BT or TT partitioning is allowed. In this case, when the CU is a subtree of BT/TT, when the size of the CU does not satisfy the width/height condition according to QT division, and when the depth of the CU does not satisfy the maximum depth of QT includes When the RD cost J _QT according to the QT division cannot be calculated, J _QT may be set as the maximum value of the RD cost.

The derivative characteristics can then be varied based on the orientation of the TT structure being tested. For example, the BSR for the TT_H division increases as the height of the block is greater than the width, and the BSR for the TT_V division increases as the width of the block is larger than the height. Therefore, in the case of TT_H partitioning, partitioning including horizontally thin blocks is not preferred. Finally, the derivative feature BTD can also be varied based on the orientation of the TT structure being tested. It can be assumed that the most appropriate direction of TT segmentation is associated with the direction of BT segmentation. For example, if RD cost J _{BT_H} is smaller than J _{BT_V} , RD cost J _{TT_H} may also be smaller than J _{TT_V} . Accordingly, the BTD value may be calculated differently depending on which of the TT_H or TT_V division is tested.

The image encoding apparatus may perform a screening test (ST) (S604), and may perform early TT segmentation according to the result. ST is defined as shown in Equation 1, and when it is 1, early TT division is performed, and when it is 0, regular TT division (S508 and S510) may be performed.

Here, TC is a tree structure channel including a luma channel (LC) and a chroma channel (CC). Also, Pic is the area of the current picture. TD (Tree Depth) is the tree depth for the MTT structure, a value included in {0,1,2,3}, when the CU violates the picture boundary (when at least one pixel is not included in the Pic) 3 It may be a larger value.

ST according to the present disclosure may be performed based on a dual tree concept and unique features of the MTT structure. Here, first, when TC=CC, since a corresponding tree may be predicted/transformed using a different coding tool, it may be difficult to determine whether TT splitting is required even in such a different situation. Therefore, in order to reduce the complexity that may occur when TC=CC, the LNN is configured only for the luma dual tree. Next, the ST may prevent an unexpected (unexpected) division of the BT_H or BT_V division regardless of the RD cost. Such unexpected division may occur at the boundary/corner of a picture when the CU violates the picture boundary, that is, when the size of the picture is not a multiple of the maximum CU. Therefore, the LNN can make a suboptimal selection according to the lack of reasonable characteristics on the input side, so that the LNN is not used when the CU violates the picture boundary. Finally, if the TD is 3 or higher, the LNN is not used. Usually, the maximum value of TD is 3, but when a CU violates a picture boundary, TD may increase due to unexpected division as described above.

Hereinafter, an early TT segmentation process performed when ST is 1 will be described.

The image encoding apparatus generates an output y _h by using the LNN _H based on the explicit characteristic and the derived characteristic (S606). Here, LNN _H is a lightweight neural network, and may be implemented by including a small number of parameters so that the amount of computation by the neural network does not burden the entire encoding process.

The video encoding apparatus compares the output y _h with a preset threshold (eg, 0.5) (S608), and if it is greater than the threshold, calculates an RD cost J _{TT_H} for TT_H division (S610).

When the output y _h is less than or equal to a preset threshold, the entire process of dividing TT_H is omitted. Here, the entire process of TT_H partitioning includes even partitioning of a child node of the CU, but there is no restriction on the tree structure of the child node. Also, J _{TT_H} may be set to the maximum value of the RD cost.

The image encoding apparatus generates an output y _v by using the LNN _V based on the explicit characteristic and the derived characteristic (S612). Here, LNN _V is a lightweight neural network.

The video encoding apparatus compares the output y _v with a preset threshold (eg, 0.5) (S614), and if it is greater than the threshold, calculates an RD cost J _{TT_V} for TT_V division (S616).

When the output y _v is less than or equal to a preset threshold, the entire process of TT_V division is omitted. Here, the entire process of TT_V partitioning includes partitioning of the child node of the CU, but there is no restriction on the tree structure of the child node. Also, J _{TT_V} may be set to the maximum value of the RD cost.

The image encoding apparatus determines an intra prediction mode and a partition structure for the CU based on the calculated cost values (S512). The cost values calculated here include J _cur , QT, BT_H and BT_V split according to intra prediction, J _QT , J _{BT_H} and J _{BT_V} , and J _{TT_H} and J _{TT_V} according to early TT split or regular TT split.

In the example of FIG. 6 , the threshold value for the output y _h and the threshold value for y _v are exemplified as the same value, but the present invention is not limited thereto and may be different values.

According to a false negative that the output of the LNN is judged to be less than or equal to the threshold, in particular, when the MTT depth is low, in order to prevent the encoding performance from being significantly degraded by not performing TT division, the threshold is set to a value lower than 0.5 (e.g., for example, 0.25 or 0.125).

Meanwhile, the lightweight neural networks LNN _H and LNN _V may be LNNs having the same structure, but may have different parameter values. The LNN is a fully connected deep neural network, and may include an input layer, a hidden layer, and an output layer. In order to accommodate the characteristic data as described above, the input layer includes 5 nodes, and the hidden layer and the output layer include 30 and 1 nodes, respectively. Accordingly, the number of parameters for each layer including the bias can be exemplified as shown in Table 3.

In the lightweight neural network, the number of parameters to be trained is 221 in total, which is a number that does not burden the increase in complexity of the image encoding apparatus. In addition, by applying a sigmoid function to the output layer, the output of the LNN is made to exist in the range from 0 to 1. As described above, when the output of LNN is 1, it indicates that TT is the optimal choice for CU in terms of RD cost. On the other hand, when the output of the LNN is 0, it indicates that it is not an optimal choice for the CU. The output of LNN located between 0 and 1 represents the probability of suitability of TT in terms of RD cost.

The bar shown in Table 3 is an example of the structure of the LNN and is not necessarily limited thereto.

An LNN (ie, LNN _H and LNN _V , respectively) may be trained based on the class to which the learning characteristic data and the target output are paired. The learning characteristic data may be obtained using the rate-distortion optimization process illustrated in FIG. 6 , based on a QP value different from that used in the actual image encoding process. The target output is a binary value, indicating whether the TT segmentation is optimal for the training feature data.

A loss function is defined based on a distance between an output of the LNN for learning characteristic data and a target output, and by updating the parameters of the LNN in a direction to decrease the loss function, training for the LNN may be performed. Here, the distance may be any one capable of expressing a distance difference between two comparison objects, such as an L1 metric and an L2 metric.

8 is a schematic block diagram of an image encoding apparatus for determining a terrestrial tree according to an embodiment of the present disclosure.

An image encoding apparatus according to the present disclosure determines a ternary tree using an LNN based on rate-distortion optimization. The video encoding apparatus includes an intra prediction unit 124 , a QT/BT cost estimation unit 802 , a second cost estimation unit 804 , and an optimal division determining unit 806 in addition to existing components including a feature extraction unit 810 . ), a screening test unit 812 and an additional component of the first cost estimator 814 .

The intra prediction unit 124 calculates the cost value J _cur for the current CU according to rate-distortion by performing intra prediction and transformation on the CU using a plurality of coding tools.

If additional division of the CU is possible, the image encoding apparatus may perform a test based on MTT division.

The QT/BT cost estimator 802 calculates costs J _QT , J _{BT_H} and J _{BT_V} according to rate-distortion by sequentially applying the QT division, the BT_H division, and the BT_V division.

The second cost estimator 804 performs TT normal division according to the result of the screening test. That is, the second cost estimator 804 calculates the costs J _{TT_H} and J _{TT_V} according to rate-distortion by sequentially applying the TT_H division and the TT_V division.

The optimal partition determination unit 806 determines an intra prediction mode and a partition structure for the CU based on the calculated costs.

The feature extractor 810 extracts a feature of an image within the CU to be used as an input of the LNN in the intra prediction, QT segmentation, and BT segmentation steps for the CU. The properties for the image may include explicit properties and derived properties, as shown in Table 1.

In order to reduce the complexity of the LNN, among the properties shown in Table 1, only some properties helpful in improving the accuracy of the LNN may be used. In the present disclosure, as shown in Table 2, BSR, BTD, BTS, MTD, and CBF are selected as final characteristics in the order in which the correlation with the binary class is significant and used as an input to the LNN. Among them, MTD and CBF are values that can be directly calculated from a coding tool and a coding parameter after intra prediction for the current CU, so they are expressed as explicit properties. Since the remaining three characteristics are values that can be calculated based on the block shape and RD cost after BT division is performed, they are expressed as derivative characteristics.

The screening test unit 812 may determine whether to perform TT early segmentation by generating a test result as shown in Equation (1).

The first cost estimator 814 performs TT early division according to the result of the screening test.

The first cost estimator 814 generates an output y _h using LNN _H based on the explicit and derived characteristics. When the output y _h is greater than the preset threshold, the RD cost J _{TT_H} for the TT_H split is calculated. When the output y _h is greater than a preset threshold, the first cost estimator 814 omits the entire process for TT_H division, and then generates an output y _v using LNN _V based on explicit and derived characteristics. . When the output y _v is greater than the preset threshold, the RD cost J _{TT_V} for the TT_V split is calculated. When the output y _v is less than or equal to a preset threshold, the entire process of TT_V division is omitted.

As described above, according to this embodiment, a ternary tree is determined during MTT (Multi-Type Tree) using a lightweight neural network based on explicit and derived characteristics extracted in a coding unit (CU) encoding process. By providing the video encoding method, it is possible to reduce the depth of the MTT structure for intra prediction and thus effectively reduce the computational complexity of the video encoding method for performing VVC.

The image encoding method for determining the ternary tree based on the lightweight neural network as described above uses intra prediction for CUs such as I frame (Intra frame), P frame (Predicted frame), and B frame (Bipredictive frame). It can be applied to all possible types of frames.

In another embodiment of the present disclosure, an image encoding method for determining a ternary tree based on a lightweight neural network may be applied even when inter prediction is used for a CU.

Although it is described that each process is sequentially executed in each flowchart according to the present embodiment, the present invention is not limited thereto. In other words, since it may be applicable to change and execute the processes described in the flowchart or to execute one or more processes in parallel, the flowchart is not limited to a time-series order.

It should be understood that the exemplary embodiments in the above description may be implemented in many different ways. The functions or methods described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein have been labeled "...unit" to particularly further emphasize their implementation independence.

Meanwhile, various functions or methods described in this embodiment may be implemented as instructions stored in a non-transitory recording medium that can be read and executed by one or more processors. The non-transitory recording medium includes, for example, all kinds of recording devices in which data is stored in a form readable by a computer system. For example, the non-transitory recording medium includes a storage medium such as an erasable programmable read only memory (EPROM), a flash drive, an optical drive, a magnetic hard drive, and a solid state drive (SSD).

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

(Explanation of symbols)

124: intra prediction unit

802: QT/BT Cost Estimator

804: second cost estimator

806: optimal division determining unit

810: feature extraction unit

812: screening test unit

814: first cost estimator

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority to Patent Application No. 10-2020-0159984, filed in Korea on November 25, 2020, and Patent Application No. 10-2021-0030727, filed in Korea on March 9, 2021 and all contents thereof are incorporated into this patent application by reference.

Claims (10)

Video encoding for performing an encoding test for determining the division type and direction of a target CU (Coding Unit) corresponding to any one node of the tree structure for a tree structure including multi-type tree (MTT)-based block division In the method, performing an encoding test on the target CU to generate encoding costs for each of non-split, QT (Quad Tree) split, and BT (Binary Tree) split; generating an explicit characteristic and a derivative characteristic for the target CU, wherein the explicit characteristic includes at least some of coding information generated according to an encoding test for the target CU, and the derivative characteristic is the encoding costs and derived based on at least one of a ternary tree (TT) division direction to be tested; and Determining whether to skip the encoding test for TT splitting in the horizontal or vertical direction by inputting the explicit and derived characteristics into one or more deep neural networks An image encoding method comprising a. According to claim 1, The step of determining whether to skip the encoding test includes: By generating an output using the deep neural network based on the explicit and derivative properties, and comparing the output of the deep neural network with a preset threshold, TT segmentation in the horizontal or vertical direction is performed and a corresponding encoding cost An image encoding method, characterized in that it is determined whether to generate According to claim 1, The explicit property is Including all or part of a Coded Block Flag (CBF) and an MTT Depth (MTD), wherein the CBF is a characteristic generated based on a flag indicating whether a transform tree structure for the target CU exists, and the MTD is the BT An image encoding method, characterized in that the characteristic is generated based on the depth of segmentation and TT segmentation. According to claim 1, The derivative property is Including all or part of Block Shape Ratio (BSR), BT Direction (BTD), and BTS' Superiority (BTS), the BSR is a characteristic generated based on the ratio of the block shape according to the direction of TT division, and the BTD is a characteristic indicating whether the optimal direction for the BT segmentation in terms of the encoding cost coincides with the direction of the TT segmentation to be tested, and the BTS determines whether the BT segmentation is An image encoding method, characterized in that it is a characteristic indicating excellence. According to claim 1, The deep neural network is An image encoding method, which is implemented as a pre-connected neural network, and is trained in advance based on characteristic data for learning and a target output. According to claim 1, Further comprising the step of generating a screening result by performing a screening test based on the structure of the dual tree (dual tree) for the target CU and the structure of the MTT, if the screening result is true, the encoding test and determining whether to skip , and when the screening result is false, performing the horizontal TT division and the vertical TT division to generate a corresponding encoding cost. 7. The method of claim 6, The step of generating the screening result comprises: If the structure of the dual tree is a luma tree, the target CU exists in a corresponding picture, and the tree depth according to the MTT for the target CU is less than or equal to a preset value, the screening result is generated as true Video encoding method, characterized in that. In performing a coding test for determining the splitting type and direction of a target CU (coding unit) corresponding to any one node of the tree structure for a tree structure including multi-type tree (MTT)-based block splitting, a CU partitioning unit generating encoding costs for each of non-split, QT (Quad Tree) split, and BT (Binary Tree) split by performing an encoding test on the target CU; A feature extracting unit generating an explicit characteristic and a derivative characteristic for the target CU, the explicit characteristic includes at least some of coding information generated according to an encoding test for the target CU, and the derivative characteristic is the encoding derived based on at least one of costs and a TT (Termary Tree) partitioning direction to be tested; and A first cost estimator that determines whether to skip the encoding test for TT splitting in the horizontal or vertical direction by inputting the explicit and derived characteristics into one or more deep neural networks An image encoding apparatus comprising a. 9. The method of claim 8, The first cost estimating unit, By generating an output using the deep neural network based on the explicit and derivative properties, and comparing the output of the deep neural network with a preset threshold, the encoding cost according to the TT segmentation in the horizontal or vertical direction is reduced An image encoding apparatus, characterized in that it is determined whether to generate the image. 9. The method of claim 8, a screening test unit for generating a screening result by performing a screening test on the target CU based on a structure of a dual tree and a structure of the MTT; and Further comprising a second cost estimating unit, When the screening result is true, the first cost estimator determines whether to skip the encoding test for the TT division in the horizontal or vertical direction, and when the screening result is false, the second cost estimator performs the horizontal and vertical A video encoding apparatus, characterized in that by performing TT segmentation in the direction, each corresponding encoding cost is generated.

Images