WO2024164590A1 - Quantization method for encoder-decoder network model and related apparatus - Google Patents

Abstract

The present application relates to the field of image compression, and discloses a quantization method and apparatus for an encoder-decoder network model, a device, a storage medium, and a computer program. The method comprises: determining H network units comprised in an unquantized encoder-decoder network model; scaling weights of output channels comprised in a first network layer in each network unit, so that the difference between boundary feature values of channels in a feature map output by the first network layer in each network unit is smaller than a feature threshold; scaling weights of input channels comprised in a second network layer in each network unit, so that output results of the encoder-decoder network model before and after the model weight scaling are the same; and quantizing the encoder-decoder network model having experienced the model weight scaling. After the encoder-decoder network model having experienced the model weight scaling is quantized, there are no abnormalities in a finally generated bitstream and/or reconstructed image, thereby improving the stability of the encoder-decoder network model.

Description

Quantization method and related device of codec network model

This application claims priority to Chinese patent application No. 202310153239.1, filed on February 8, 2023, and entitled “Quantization method and related device for coding and decoding network model”, the entire contents of which are incorporated by reference into this application.

Technical Field

The present application relates to the field of image compression, and in particular to a quantization method and related device for a coding and decoding network model.

Background Art

With the widespread application of deep learning technology in image recognition, target detection and other fields, deep learning technology has also been applied to image compression tasks, that is, using codec network models for image compression. This method of using codec network models for image compression is superior to traditional image compression methods in terms of codec efficiency and image compression effect. For example, using variational auto-encoder (VAE) for image encoding and decoding can greatly improve the codec performance and image compression effect. However, the codec network model introduces a huge amount of parameters and calculations while improving the codec performance, which leads to various problems when the codec network model runs on resource-limited end-side devices. For example, when the codec network model runs on a low-performance mobile device or a low-power embedded device, the efficiency of model inference will be reduced. For another example, some end-side devices do not support floating-point calculations of the codec network model, which limits the deployment of the codec network model on the end-side device.

In order to solve the above problems, model quantization came into being. Model quantization is a technology that converts the floating-point calculation of the model into fixed-point calculation, which can effectively reduce the model calculation amount, parameter size and memory consumption. After quantization, the codec network model can be deployed on end-side devices with limited resources such as mobile phones and robots. However, the above codec network model has the problem of poor stability. For example, the bit rate after encoding the image is too high, and there are anomalies in the reconstructed image. Even if the codec network model is quantized, the problem of poor stability of the codec network model still exists.

Summary of the invention

The present application provides a method, device, equipment, storage medium and computer program for quantizing a codec network model, which can solve the problem of poor stability of the codec network model in the related art. The technical solution is as follows:

In a first aspect, a quantization method for a codec network model is provided, the method comprising: determining H network units included in an unquantized codec network model, each of the H network units comprising a first network layer, a second network layer and a third network layer, the first network layer and the second network layer are both linear operation layers, the third network layer is located between the first network layer and the second network layer, the third network layer is used to truncate the eigenvalues in the feature map output by the first network layer, so that the eigenvalues in the feature map output by the third network layer are all non-negative or all non-positive, H is an integer greater than or equal to 1, and for each network The weights of each output channel included in the first network layer in the unit are scaled so that the difference between the boundary feature values of each channel in the feature map output by the first network layer in each network unit is less than the feature threshold, and the weights of each input channel included in the second network layer in each network unit are scaled so that the output result of the codec network model is the same before and after the model weight scaling. The model weight scaling includes weight scaling of each output channel of the first network layer in each network unit and weight scaling of each input channel of the second network layer. The codec network model after the model weight scaling is quantized.

By scaling the weights of each output channel included in each first network layer, it is possible to ensure that the difference between the boundary feature values of each channel in the feature map output by each first network layer is less than the feature threshold, and by scaling the weights of the input channels of the second network layer after the first network layer, it is possible to ensure that the output result of the codec network model is the same before and after the model weight scaling. In this way, after the codec network model after the model weight scaling is quantized, the first network layer and the second network layer after the model weight scaling can truncate the abnormal feature values, and can also ensure that the normal feature values are not affected by the model weight scaling, thereby reducing the impact of the codec network model on the weight change of each output channel of the first network layer, so that the final generated code stream and/or reconstructed image does not have abnormalities, and improves the stability of the codec network model.

It should be noted that an image can output a feature map after passing through a network layer. The codec network model contains multiple network layers, and each network layer includes multiple weights, which are the weights of the input channel and the output channel of the corresponding network layer. The weights of can process the feature maps of the network layer input, and the weights of the output channels can process the feature maps of the network layer output. The image to be compressed is input into the codec network model, and each network layer in the codec network model can output the corresponding feature map.

Optionally, determining a first network layer among H network units included in an unquantized codec network model includes: determining M first sample feature maps output by a first linear operation layer, the M first sample feature maps are feature maps corresponding to M normal sample images, M is an integer greater than or equal to 1, the first linear operation layer is any linear operation layer included in the codec network model, determining N second sample feature maps output by the first linear operation layer, the N second sample feature maps are feature maps corresponding to N abnormal sample images, N is an integer greater than or equal to 1, and when the M first sample feature maps and the N second sample feature maps meet a feature abnormality condition, determining that the first linear operation layer is the first network layer.

Since not every linear operation layer in the codec network model will cause abnormal coding results, the linear operation layers included in the codec network model can also be screened to determine the linear operation layers that will cause abnormal coding results, and the linear operation layers that will cause abnormal coding results are used as the first network layer. In this way, the weights of each output channel included in the linear operation layer that causes abnormal coding results can be scaled in a targeted manner, which can reduce the computational complexity of the codec network model while effectively ensuring the stability of the codec network model.

Optionally, determining the second network layer in the H network units included in the unquantized codec network model includes: for the first network layer in each of the network units, when there is a linear operation layer after the first network layer, determining the linear operation layer after the first network layer as the second network layer; when there is no linear operation layer after the first network layer, adding a linear operation layer after the first network layer, and determining the added linear operation layer as the second network layer.

Since the second network layer is a linear operation layer after the first network layer, the second network layer can timely correct the feature values in the feature map output by the first network layer, effectively reducing the impact of the codec network model caused by the change of the weights of each output channel of the first network layer, so that the output result of the codec network model is the same before and after the model weight scaling, thereby ensuring that there is no abnormality in the final generated code stream and/or reconstructed image, and improving the stability of the codec network model.

Optionally, scaling the weights of each output channel included in the first network layer in each network unit includes: for the first network layer in each network unit, determining K third sample feature maps output by the first network layer, the K third sample feature maps are feature maps corresponding to K normal sample images, K is an integer greater than or equal to 1, determining a scaling ratio of each output channel included in the first network layer based on the K third sample feature maps, and scaling the weights of each output channel included in the first network layer according to the scaling ratio of each output channel included in the first network layer.

Since the normal sample image is an image without any abnormalities in the bit rate and the reconstructed image after passing through the codec network, and the value range of the eigenvalues in each channel of the feature map corresponding to the normal sample image is smaller than the value range of the eigenvalues in the corresponding channel of the feature map corresponding to the abnormal sample image, therefore, by scaling the weights of each output channel included in the first network layer based on the scaling ratio of the output channels determined by the third sample feature map, the effective truncation of the abnormal eigenvalues can be achieved, thereby ensuring that the values of the truncated abnormal eigenvalues are consistent with the values of the eigenvalues in the normal sample feature map, thereby improving the stability of the codec network model.

Optionally, determining the scaling ratio of each output channel included in the first network layer based on the K third sample feature maps includes: determining a reference eigenvalue corresponding to the first network layer and a boundary eigenvalue of each output channel included in the first network layer based on the K third sample feature maps, and determining the ratio between the reference eigenvalue corresponding to the first network layer and the boundary eigenvalue of each output channel included in the first network layer as the scaling ratio of each output channel included in the first network layer.

Since the range of eigenvalues of the feature map corresponding to the normal sample image in each channel is smaller than the range of eigenvalues of the feature map corresponding to the abnormal sample image in the corresponding channel, the boundary eigenvalues of each output channel included in the first network layer determined based on the third sample feature map can indicate the range of eigenvalues of the feature map corresponding to the normal sample image in each channel, and the reference eigenvalues corresponding to the first network layer can indicate the maximum eigenvalue in the feature map corresponding to the normal sample image. In this case, the ratio between the reference eigenvalues corresponding to the first network layer and the boundary eigenvalues of each output channel included in the first network layer is determined as the scaling ratio of each output channel included in the first network layer, which can ensure the truncation of abnormal eigenvalues and can also ensure that normal eigenvalues are not affected by the model weight scaling, thereby effectively improving the stability of the codec network model.

Optionally, scaling the weights of the respective input channels included in the second network layer in each network unit includes: for the second network layer in each network unit, determining the scaling ratio of the respective input channels included in the second network layer based on the scaling ratio of the respective output channels included in the first network layer in the network unit where the second network layer is located, and scaling the weights of the respective input channels included in the second network layer according to the scaling ratio of the respective input channels included in the second network layer.

According to the scaling ratio of each input channel included in the second network layer, the weights of each input channel included in the second network layer are scaled, so that the second network layer can modify the feature values in the feature map output by the corresponding first network layer in a targeted manner, so that the normal feature The eigenvalue is not affected by the weight scaling of the first network layer, thereby ensuring that the output results of the normal sample image in the codec network model are the same before and after the model weight scaling, thereby improving the stability of the codec network model.

Optionally, the encoding and decoding network model includes an encoding network model, a decoding network model or an entropy estimation network model.

Regardless of whether the codec network model includes an encoding network model, a decoding network model or an entropy estimation network model, the quantization method of the codec network model provided in the embodiment of the present application can determine the first network layer and the second network layer from the codec network model, and then realize the correction of abnormal feature values, so that the finally generated bitstream and/or reconstructed image does not have abnormalities, thereby improving the stability of the codec network model.

Optionally, the boundary eigenvalue is a maximum eigenvalue or a minimum eigenvalue.

In the case where the boundary eigenvalue is the maximum eigenvalue, that is, when the abnormal eigenvalue greater than the maximum truncation value is truncated, the minimum eigenvalue in the feature map output by the third network layer is 0, that is, all are non-negative. In this case, since the minimum truncation value is generally a negative number, the abnormal eigenvalue less than the minimum truncation value is truncated to 0 after passing through the third network layer, thereby achieving the correction of the abnormal eigenvalue. In the case where the boundary eigenvalue is the minimum eigenvalue, the maximum eigenvalue in the feature map output by the third network layer is 0, that is, all are non-positive. In this case, since the maximum truncation value is generally a positive number, the abnormal eigenvalue greater than the maximum truncation value is truncated to 0 after passing through the third network layer, thereby achieving the correction of the abnormal eigenvalue. In other words, regardless of whether the boundary eigenvalue is the maximum eigenvalue or the minimum eigenvalue, the quantization method of the codec network model provided in the embodiment of the present application can realize the correction of the abnormal eigenvalue, so that the final generated code stream and/or reconstructed image does not have abnormalities, thereby improving the stability of the codec network model.

Among them, the value range of the feature value in the normal sample feature map output by any network layer in the codec network model is called the truncation value of the network layer, and the truncation value includes the minimum truncation value and the maximum truncation value. The minimum truncation value refers to the minimum value in the value range, and the maximum truncation value refers to the maximum value in the value range.

In a second aspect, a quantization device for a codec network model is provided, wherein the quantization device for the codec network model has the function of implementing the quantization method behavior of the codec network model in the first aspect. The quantization device for the codec network model includes at least one module, and the at least one module is used to implement the quantization method of the codec network model provided in the first aspect.

In a third aspect, a quantization device for a codec network model is provided, the quantization device for the codec network model comprising a processor and a memory, the memory being used to store a computer program for executing the quantization method for the codec network model provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the quantization method for the codec network model described in the first aspect.

Optionally, the quantization device of the codec network model may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.

In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the quantization method of the encoding and decoding network model described in the first aspect.

In a fifth aspect, a computer program product comprising instructions is provided, and when the instructions are executed on a computer, the computer executes the quantization method of the codec network model described in the first aspect. In other words, a computer program is provided, and when the computer program is executed on a computer, the computer executes the quantization method of the codec network model described in the first aspect.

The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an image compression framework provided in an embodiment of the present application;

FIG2 is a schematic diagram of a network model provided in an embodiment of the present application;

FIG3 is a statistical diagram of a maximum eigenvalue provided in an embodiment of the present application;

FIG4 is a statistical diagram of a minimum eigenvalue provided in an embodiment of the present application;

FIG5 is a statistical diagram of another maximum eigenvalue provided in an embodiment of the present application;

FIG6 is a schematic diagram of a reconstructed image provided by an embodiment of the present application;

FIG7 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

FIG8 is a schematic diagram of another implementation environment provided by an embodiment of the present application;

FIG9 is a schematic diagram of the structure of a coding and decoding framework provided in an embodiment of the present application;

FIG10 is a schematic diagram of the structure of a coding network model provided in an embodiment of the present application;

FIG11 is a schematic diagram of the structure of a decoding network model provided in an embodiment of the present application;

FIG12 is a schematic diagram of the structure of an entropy estimation network model provided in an embodiment of the present application;

FIG13 is a flowchart of a quantization method of a coding and decoding network model provided in an embodiment of the present application;

FIG14 is a schematic diagram of a network unit provided in an embodiment of the present application;

FIG15 is a schematic diagram of a weight provided in an embodiment of the present application;

FIG16 is a statistical diagram of another maximum eigenvalue provided in an embodiment of the present application;

FIG17 is a schematic diagram of another weight provided in an embodiment of the present application;

FIG18 is a schematic diagram of another reconstructed image provided by an embodiment of the present application;

FIG19 is a schematic diagram of the structure of an entropy estimation network model provided in an embodiment of the present application;

FIG20 is a schematic diagram of the structure of a quantization device for a coding and decoding network model provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

For ease of understanding, before explaining in detail the quantization method of the encoding and decoding network model provided in the embodiment of the present application, the nouns, application scenarios and implementation environment involved in the embodiment of the present application are first introduced.

First, the terms involved in the embodiments of the present application are explained.

Artificial intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

Rectified linear unit (ReLU): is a commonly used activation function in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

Convolutional neural network (CNN): It is a feedforward neural network with deep structure and convolution calculation. It is one of the representative algorithms of deep learning. CNN may also contain activation layer (such as ReLU, etc.), pooling layer, batch normalization layer, fully connected layer and other modules. Typical convolutional neural networks include LeNet, AlexNet, VGGNet, ResNet, etc. Basic CNN can be composed of backbone network and head network, and complex CNN is composed of backbone, neck and head network.

Feature map: refers to the three-dimensional data output by the network layers such as convolution layer, activation layer, pooling layer, batch normalization layer, etc. in CNN. The three dimensions of this three-dimensional data are width, height, and channel.

In an embodiment of the present application, an image can output a feature map after passing through a network layer. For any network layer, if the network layer contains three convolution kernels, the three convolution kernels can respectively convolve the feature map input by the network layer to obtain the feature map output by the network layer. The number of channels of the feature map output by the network layer is 3, that is, the number of convolution kernels contained in the network layer corresponds one to one to the number of channels of the feature map output by the network layer.

Variational Autoencoder: An AI-based image codec for data compression or noise removal. The embodiment of the present application is introduced by taking the VAE-based image compression framework as an example. The VAE-based image compression framework is shown in Figure 1. During the encoding process, the image to be compressed is input into the encoding network model to obtain the features to be encoded. The features to be encoded are quantized to obtain the quantized features to be encoded. Based on the quantized features to be encoded, the probability distribution is estimated through the entropy estimation network model. The quantized features to be encoded are entropy encoded based on the probability distribution to obtain an image code stream. The decoding process corresponds to the encoding process.

Peak signal to noise ratio (PSNR) is an objective standard for evaluating image quality. The higher the PSNR, the better the image quality.

Bit rate: In image compression, it refers to the code length required to encode a unit pixel. The higher the bit rate, the better the image reconstruction quality.

Pixel bit rate (bits per pixel, BPP): also known as bits/pixel, BPP is the number of bits used to store each pixel. The smaller the BPP, the lower the compression bit rate, and the larger the BPP, the higher the compression bit rate.

Quantization: refers to converting a continuous signal into a discrete signal. In the image compression process, it refers to converting a continuous feature into a discrete signal. In entropy coding, the probability value of the probability distribution is usually changed from a continuous value to a discrete value.

Rate-distortion curve (RD-Curve): The horizontal axis is the bit rate, and the vertical axis is the image quality. Generally, the closer the curve is to the upper left, the better the encoding and decoding performance.

AI compression algorithm (AICodec): refers to a data compression method based on deep learning technology.

Entropy estimation: The process of predicting the distribution of quantized data.

Model quantization: refers to the method of converting a floating-point neural network model into a fixed-point model that can be represented by a finite number of bits. Model quantization can establish a mapping relationship between floating-point data and fixed-point data. Its principle can be understood as using integers in the range of n (e.g., 8, 10, 16) bits to represent the floating-point weights in the network model and the tensors in the calculation process. For ease of understanding, the process of quantizing tensors in the model calculation process is introduced by taking the network model shown in Figure 2 as an example. In Figure 2, x represents the input of the network model, y represents the output of the network model, a1 is the feature map output by the convolutional layer (Conv), a2 is the feature map output by the activation layer (Relu), qx represents the fixed-point value of x after quantization, qa1 represents the feature map after quantization of a1, qa2 represents the feature map after quantization of a2, and qy represents the data output by the fully connected layers (FC). In model quantization, based on sample data, the value ranges of x, a1, a2, and y can be statistically analyzed to obtain the maximum and minimum values in the value ranges of x, a1, a2, and y. Based on the maximum and minimum values in the value ranges of x, a1, a2, and y, the quantization formulas corresponding to x, a1, a2, and y are determined. In this way, during the inference process of the model, the input of the model and the tensor in the calculation process can be quantized based on the quantization formulas corresponding to x, a1, a2, and y to obtain qy, and qy can be dequantized to obtain the inferred floating-point value y.

As an example, the quantization formula corresponding to the input x of the network model can be expressed by the following expression (1).

Among them, in the above expression (1), x _quant represents the fixed-point value after quantization, x represents the floating-point value that has not been quantized, scale represents the minimum scale that the fixed-point value after quantization can represent, that is, the scale, zero_point represents the fixed-point value corresponding to the floating-point value 0, that is, the zero point position, quant _min represents the minimum value in the value range of x _quant , quant _max represents the maximum value in the value range of x _quant , the round() function is used to round the numbers in the brackets, and the clamp() function is used to limit x _quant to between quant _max and quant _min . That is, if the calculated fixed-point value exceeds the value range of x _quant , it needs to be truncated so that x _quant is equal to the value of quant _max and quant _min that is closest to the calculated fixed-point value.

The main model quantization methods include post-training quantization (PTQ) and quantization aware training (QAT). Among them, PTQ refers to directly converting a floating-point network model into a fixed-point network model without retraining the network model, that is, without updating the weights of the network model. QAT refers to inserting a pseudo-quantization (simulated quantization) operation into the network model during network model training or fine-tuning to simulate the quantization operation, so that the parameters of the network model can better adapt to the information loss caused by quantization. Usually, applying QAT to the fine-tuning process of the network model, fine-tuning from a pre-trained network model can make the network model more accurate and robust, and can significantly reduce the number of iterations of network model training.

Image compression refers to a technology that uses image data characteristics such as spatial redundancy, visual redundancy, and statistical redundancy to represent the original image pixel matrix losslessly or with fewer bits, achieving effective transmission and storage of image information. It plays an important role in the current media era where the types and amount of image information are increasing. Image compression technology includes encoding and decoding of images, and encoding and decoding performance (reflecting image quality) and encoding and decoding efficiency (reflecting time consumption) are factors that need to be considered in image compression technology.

After long-term research and optimization by technical personnel, lossy image compression standards such as the Joint Photographic Experts Group (JPEG) have been formed. However, these more traditional image compression technologies have encountered bottlenecks in improving encoding and decoding performance, and can no longer meet the needs of the era of increasing multimedia application data. With the widespread application of deep learning technology in image recognition, target detection and other fields, deep learning technology has also been applied to image compression tasks, that is, using codec network models for image compression. This method of using codec network models for image compression is superior to traditional image compression methods in terms of encoding and decoding efficiency and image compression effect. For example, using variational autoencoders for image encoding and decoding can greatly improve encoding and decoding performance and image compression effect.

In practical applications, while the encoding and decoding efficiency and image compression effect of AI compression algorithms are improved, there are also high requirements for the stability of AI compression algorithms. Stability mainly includes two aspects. On the one hand, it is the stability of the bit rate after encoding the image without adding noise and the stability of the effect of reconstructing the image. On the other hand, it is the stability of the bit rate after encoding the image with specific noise and the stability of the effect of reconstructing the image, that is, the stability against attacks. However, AI compression algorithms have poor stability in both aspects. For example, some The bit rate of the image with added noise after passing through the codec network model of the AI compression algorithm is too high and/or the reconstructed image has abnormalities. For another example, the bit rate of the image with added specific noise increases abnormally and/or the reconstructed image has abnormalities after passing through the codec network model, that is, the performance of the AI compression algorithm is severely reduced. For the sake of ease of description, in the following text, images with no abnormalities in bit rate and reconstructed image after passing through the codec network model are referred to as normal sample images, and images with abnormalities in bit rate and/or reconstructed image after passing through the codec network model are referred to as abnormal sample images.

In order to clarify the reasons why the bit rate and/or reconstructed image of the abnormal sample image is abnormal after passing through the codec network model, the technicians have counted the feature maps output by the normal sample image and the abnormal sample image in each layer of the codec network model. The feature map output by any layer of the normal sample image in the codec network model can be called a normal sample feature map, and the feature map output by any layer of the abnormal sample image in the codec network model can be called an abnormal sample feature map. That is, the maximum eigenvalue and the minimum eigenvalue in the feature map of each channel included in the normal sample feature map and the abnormal sample feature map are counted. Based on the statistical data, the technicians found that the maximum eigenvalue and the minimum eigenvalue in the feature map of each channel included in the normal sample feature map and the abnormal sample feature map output in the same network layer are very different. Please refer to FIG3. Taking the hyper encoder (HyEnc) network model included in the encoding and decoding network model as an example, it can be seen from the statistical graph of the maximum eigenvalue in the feature graph of each channel included in the normal sample feature graph and the abnormal sample feature graph outputted by the first network layer in the hyperencoding network model that in the first network layer in the hyperencoding network model, the maximum eigenvalue in the feature graph of each channel included in the abnormal sample feature graph is greater than the maximum eigenvalue in the feature graph of each channel included in the normal sample feature graph. Please refer to FIG4. It can be seen from the statistical graph of the minimum eigenvalue in the feature graph of each channel included in the normal sample feature graph and the abnormal sample feature graph outputted by the first network layer in the hyperencoding network model that in the first network layer in the hyperencoding network model, the minimum eigenvalue in the feature graph of each channel included in the abnormal sample feature graph is less than the minimum eigenvalue in the feature graph of each channel included in the normal sample feature graph. In other words, the range of eigenvalues in the feature graph of each channel of the normal sample feature graph is smaller than the range of eigenvalues in the feature graph of the corresponding channel of the abnormal sample feature graph.

In addition, the codec network model introduces a huge amount of parameters and calculations while improving the codec performance, which leads to various problems when the codec network model runs on a resource-limited end-side device. For example, when the codec network model runs on a low-performance mobile device or a low-power embedded device, the efficiency of model reasoning will be reduced. For another example, some end-side devices do not support floating-point calculations of the codec network model, which limits the deployment of the codec network model on the end-side device. In other words, the above-mentioned codec network model has the problem of poor stability and is difficult to deploy on end-side devices with limited resources.

In order to solve the problem that the codec network model is difficult to deploy on end-side devices with limited resources, technicians quantize the model and convert the model's floating-point calculations into fixed-point calculations, thereby effectively reducing the model's calculation amount, parameter size, and memory consumption. This allows the codec network model to be deployed on end-side devices with limited resources, such as mobile phones and robots, after quantization.

Although the range of eigenvalues in the normal sample feature map output by each network layer in the codec network model can be determined based on the normal sample image during the model quantization process, and then when the model is inferred, the eigenvalue of each feature point in the feature map output by the corresponding network layer can be truncated based on the range of eigenvalues in the normal sample feature map output by each network layer in the codec network model, so that the value of the eigenvalue is consistent with the eigenvalue in the normal sample feature map, thereby reducing abnormal eigenvalues to a certain extent, but this cannot effectively solve the problem of poor stability of the codec network model. For the sake of convenience of description, the range of eigenvalues in the normal sample feature map output by any network layer in the codec network model is referred to as the truncation value of the network layer in the following text, and the truncation value includes the minimum truncation value and the maximum truncation value. The minimum truncation value refers to the minimum value in the value range, and the maximum truncation value refers to the maximum value in the value range. For the eigenvalue in the feature map of any channel, if the eigenvalue obviously exceeds the range of eigenvalues of the normal sample feature map in the feature map of the corresponding channel, the eigenvalue is called an abnormal eigenvalue.

For any network layer in the codec network model, the truncation value used by the network layer is the maximum eigenvalue and the minimum eigenvalue of all eigenvalues included in the normal sample feature map output by the network layer. The truncation value may be greater than or equal to the value range of the eigenvalues of the abnormal sample feature map in the feature map of a certain channel. In this case, the abnormal eigenvalues of the channel cannot be truncated. In this way, the bit rate and/or reconstructed image of the abnormal sample image after passing through the codec network model will still be abnormal, thereby failing to effectively solve the problem of poor stability of the codec network model. Please refer to Figure 5, which is a statistical graph of the maximum eigenvalues in the feature map of each channel included in the normal sample feature map and the abnormal sample feature map output by a network layer in the codec network model. The maximum truncation value of the network layer is greater than the maximum eigenvalue in the feature map of each channel included in the abnormal sample feature map. At this time, based on the maximum truncation value, the abnormal eigenvalues of other channels except channel 8 cannot be truncated. Please refer to Figure 6. The reconstructed image a in Figure 6 is the reconstructed image obtained by the abnormal sample image after the unquantized codec network model, and the reconstructed image b is the reconstructed image obtained by the abnormal sample image after the quantized codec network model. It can be seen from Figure 6 that the compression rate and reconstructed image of the abnormal sample image after the quantized codec network model are abnormal, and the compression rate and reconstructed image of the abnormal sample image after the quantized codec network model still have large abnormalities. Based on this, the embodiment of the present application provides a codec A quantization method for a network model, after quantizing the codec network model by this method, can effectively truncate the abnormal feature values of the feature map output by the first network layer in each channel, thereby correcting the feature map output by the first network layer, so that the finally generated bitstream and/or reconstructed image does not have abnormalities, thereby improving the stability of the codec network model.

Next, the implementation environment involved in the embodiments of the present application is introduced.

Please refer to FIG. 7, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. Among them, the source device 10 can generate an encoded image. Therefore, the source device 10 can also be referred to as an image encoding device. The destination device 20 can decode the encoded image generated by the source device 10. Therefore, the destination device 20 can also be referred to as an image decoding device. The link 30 can receive the encoded image generated by the source device 10, and can transmit the encoded image to the destination device 20. The storage device 40 can receive the encoded image generated by the source device 10, and can store the encoded image. Under such conditions, the destination device 20 can directly obtain the encoded image from the storage device 40. Alternatively, the storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded image generated by the source device 10. Under such conditions, the destination device 20 can stream or download the encoded image stored in the storage device 40.

The source device 10 and the destination device 20 may each include one or more processors and a memory coupled to the one or more processors, the memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer, etc. For example, the source device 10 and the destination device 20 may each include a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, a smart car machine, a smart TV, a smart speaker, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called "smart" phone, a television, a camera, a display device, a digital media player, a video game console, a car computer, or the like.

The link 30 may include one or more media or devices capable of transmitting the encoded image from the source device 10 to the destination device 20. In one possible implementation, the link 30 may include one or more communication media that enable the source device 10 to send the encoded image directly to the destination device 20 in real time. In an embodiment of the present application, the source device 10 may modulate the encoded image based on a communication standard, which may be a wireless communication protocol, etc., and may send the modulated image to the destination device 20. The one or more communication media may include wireless and/or wired communication media, for example, the one or more communication media may include a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, which may be a local area network, a wide area network, or a global network (e.g., the Internet), etc. The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the source device 10 to the destination device 20, etc., and the embodiment of the present application does not specifically limit this.

In a possible implementation, the storage device 40 may store the received encoded image sent by the source device 10, and the destination device 20 may directly obtain the encoded image from the storage device 40. Under such conditions, the storage device 40 may include any of a variety of distributed or locally accessible data storage media, for example, any of the multiple distributed or locally accessible data storage media may be a hard disk drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded images.

In one possible implementation, the storage device 40 may correspond to a file server or another intermediate storage device that can store the encoded image generated by the source device 10, and the destination device 20 may stream or download the image stored in the storage device 40. The file server may be any type of server that can store the encoded image and send the encoded image to the destination device 20. In one possible implementation, the file server may include a network server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive, etc. The destination device 20 may obtain the encoded image through any standard data connection (including an Internet connection). Any standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of the two suitable for obtaining the encoded image stored on the file server. The transmission of the encoded image from the storage device 40 may be streaming transmission, download transmission, or a combination of the two.

The implementation environment shown in FIG7 is only one possible implementation method, and the technology of the embodiment of the present application is not only applicable to the source device 10 that can encode images and the destination device 20 that can decode the encoded images shown in FIG7, but also to other devices that can encode images and decode encoded images, and the embodiment of the present application does not make specific limitations on this.

In the implementation environment shown in FIG7 , the source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, the output interface 140 may include a modulator/demodulator (modem) and/or a transmitter, where the transmitter may also be referred to as a transmitter. The data source 120 may include an image capture device (e.g., a camera, etc.), an archive containing previously captured images, a feed interface for receiving images from an image content provider, and/or a computer graphics system for generating images, or a combination of these sources of images.

The data source 120 may send an image to the encoder 100, and the encoder 100 may encode the image received from the data source 120 to obtain an encoded image. The encoder may send the encoded image to the output interface. In some embodiments, the source device 10 directly sends the encoded image to the destination device 20 via the output interface 140. In other embodiments, the encoded image may also be stored in the storage device 40 for the destination device 20 to obtain and use for decoding and/or display later.

In the implementation environment shown in FIG. 7 , the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and/or a modem. The input interface 240 may receive an encoded image via the link 30 and/or from the storage device 40, and then send it to the decoder 200, and the decoder 200 may decode the received encoded image to obtain a decoded image. The decoder may send the decoded image to the display device 220. The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20. In general, the display device 220 displays the decoded image. The display device 220 may be any of a variety of types of display devices, for example, the display device 220 may be a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

Although not shown in FIG. 7 , in some aspects, encoder 100 and decoder 200 may be integrated with an encoder and decoder, respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or in separate data streams. In some embodiments, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP), if applicable.

The encoder 100 and the decoder 200 may each be any of the following circuits: one or more microprocessors, digital signal processors (digital signal processing, DSP), application specific integrated circuits (application specific integrated circuits, ASIC), field-programmable gate arrays (field-programmable gate arrays, FPGA), discrete logic, hardware, or any combination thereof. If the technology of the embodiment of the present application is partially implemented in software, the device may store instructions for the software in a suitable non-volatile computer-readable storage medium, and may use one or more processors to execute the instructions in hardware to implement the technology of the embodiment of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be regarded as one or more processors. Each of the encoder 100 and the decoder 200 may be included in one or more encoders or decoders, and any of the encoders or decoders may be integrated as part of a combined encoder/decoder (codec) in a corresponding device.

The embodiments of the present application may generally refer to the encoder 100 as "signaling" or "sending" certain information to another device, such as the decoder 200. The term "signaling" or "sending" may generally refer to the transmission of syntax elements and/or other data used to decode the compressed image. This transmission may occur in real time or near real time. Alternatively, this communication may occur over a period of time, such as when the syntax elements are stored in the encoded bitstream to a computer-readable storage medium at the time of encoding, and the decoding device may then retrieve the syntax elements at any time after the syntax elements are stored to this medium.

The quantization method of the codec network model provided in the embodiment of the present application can be applied to a variety of scenarios. The images encoded and decoded in various scenarios can be images included in image files or images included in video files.

Fig. 8 is a schematic diagram of another implementation environment provided by an embodiment of the present application. The implementation environment includes an encoding end and a decoding end, the encoding end includes an AI encoding module, an entropy encoding module and a file saving module, and the decoding end includes a file loading module, an entropy decoding module and an AI decoding module.

During the compression process, the encoder obtains the image to be compressed, such as videos and pictures captured by the camera. After that, the AI encoding module obtains the features to be encoded and the corresponding probability distribution. Based on the probability distribution, the entropy encoding module performs entropy encoding on the features to be encoded to obtain a bitstream file, which is saved by the file saving module to obtain a compressed file of the image. The compressed file is input to the decoder, which loads the compressed file through the file loading module and obtains the reconstructed image through the entropy decoding module and the AI decoding module.

The above encoding and decoding process can be implemented by an encoding and decoding network model, and the quantization method of the encoding and decoding network model provided in the embodiment of the present application can quantize the encoding and decoding network model. Optionally, the encoding and decoding network model includes at least one network model of an encoding network model, a decoding network model and an entropy estimation network model.

It should be noted that the image encoding and decoding principles of the image compression framework shown in Figure 8 and Figure 1 are similar. The AI encoding unit in Figure 8 is equivalent to including the encoding network model and entropy estimation network model in Figure 1, and the AI decoding unit is equivalent to including the decoding network model in Figure 1. and entropy estimation network models.

Optionally, the data processing process of the AI encoding module and the AI decoding module is implemented on an embedded neural network processing unit (NPU) to improve data processing efficiency, and the processes such as entropy coding, saving files, and loading files are implemented on a central processing unit (CPU).

Optionally, the encoding end and the decoding end are one device, or the encoding end and the decoding end are two independent devices. If the encoding end and the decoding end are one device, the encoding and decoding network model model quantized by the method provided in the embodiment of the present application can be deployed on the device. If the encoding end and the decoding end are two independent devices, the encoding and decoding network model quantized by the method provided in the embodiment of the present application can be deployed in the two devices respectively. That is, for one device, the device has both an image compression function and an image decompression function, or the device has an image compression function or an image decompression function.

The quantization method of the encoding and decoding network model provided in the embodiment of the present application can be applied to a variety of scenarios, such as cloud storage, video surveillance, live broadcast, transmission and other business scenarios, and can be specifically applied to terminal recording, video albums, cloud storage, etc.

It should be noted that the quantization method of the codec network model provided in the embodiment of the present application can be applied to any codec network model. Next, the codec network model based on VAE is introduced.

Please refer to FIG9 . At the encoding end, the original image is input into the encoding network model to extract features to obtain image features y to be quantified of multiple feature points, and the image features y to be quantified of the multiple feature points are quantized to obtain the first image features of the multiple feature points. The first image feature of the plurality of feature points Input into the super coding network model to obtain the super prior features z of the multiple feature points to be quantified, and quantize the super prior features z of the multiple feature points to be quantified to obtain the first super prior features of the multiple feature points The first super-prior features of the multiple feature points are calculated according to the specified probability distribution. Entropy coding is performed to convert As shown in Figure 2 The bit sequence obtained by entropy coding is a partial bit sequence included in the code stream. This partial bit sequence (as shown by the black and white bars on the right side of FIG. 2 ) can be called a super-prior bit stream.

In addition, the first super prior features of the multiple feature points Input the super decoding network model to obtain the prior features ψ of the multiple feature points. The context model (CM) is input to obtain the context features φ of the multiple feature points. The probability distribution N(μ, σ) of the multiple feature points is estimated through a probability distribution estimation network model (shown as a gather model, GM) based on the probability distribution N(μ, σ) of the multiple feature points. The first image feature of each feature point in the multiple feature points is sequentially converted into the image feature vector. To be encoded into the code stream. The bit sequence obtained by entropy coding is a partial bit sequence included in the code stream. This partial bit sequence (as shown by the black and white bars on the left side of FIG. 9 ) can be called an image bit stream.

At the decoding end, first, the first super-prior feature of the plurality of feature points is obtained by entropy decoding from the super-prior bit stream included in the bit stream according to the specified probability distribution. The first super prior features of the multiple feature points Input the super decoding network model to obtain the prior features ψ of the multiple feature points. For the first feature point among the multiple feature points, estimate the probability distribution of the first feature point based on the prior features of the first feature point, and parse the first image feature of the first feature point from the image bit stream included in the code stream based on the probability distribution of the first feature point. For the non-first feature point among the multiple feature points, such as the first feature point, determine the surrounding information of the first feature point from the first image features of each decoded feature point, input the surrounding information of the first feature point into the context model CM to obtain the context feature of the first feature point, combine the prior features and context features of the first feature point, estimate the probability distribution of the first feature point through the probability distribution estimation network model GM, and parse the first image feature of the first feature point from the image bit stream included in the code stream based on the probability distribution of the first feature point. Entropy decode the first image feature of the multiple feature points from the code stream Afterwards, Input the decoding network model to obtain the reconstructed image.

It should be noted that if any of the above-mentioned probability distribution estimation network models uses a Gaussian model (such as a single Gaussian model or a mixed Gaussian model) to model, the estimated probability distribution parameters include mean and variance. If any of the above-mentioned probability distribution estimation network models uses a Laplace distribution model to model, the estimated probability distribution parameters include location parameters and scale parameters. If any of the above-mentioned probability distribution estimation network models uses a logistic distribution model to model, the estimated probability distribution parameters include mean and scale parameters. In addition, the probability distribution estimation network model in the embodiment of the present application can also be referred to as a factor entropy model, and the probability distribution estimation network model is a part of the entropy estimation network model, and the entropy estimation network model also includes the above-mentioned super encoding network model and super decoding network model.

FIG10 is a schematic diagram of the structure of a coding network model provided in an embodiment of the present application. The coding network model is a convolutional neural network model, which includes four convolutional layers (Conv) and three activation layers (such as Relu or other activation function structures) interspersed in cascade. The activation layer is constructed). The convolution kernel size of each convolution layer is 5×5, the number of channels of the output feature map is M, and each convolution layer downsamples the width and height by 2 times. It should be noted that the structure of the encoding network model shown in Figure 10 is not used to limit the embodiments of the present application. For example, the convolution kernel size, the number of channels of the feature map, the downsampling multiple, the number of downsampling times, the number of convolution layers, etc. can all be adjusted.

Figure 11 is a structural diagram of a decoding network model provided in an embodiment of the present application. The decoding network model is a convolutional neural network model, which includes four convolutional layers (Conv) and three activation layers interspersed in cascade (such as activation layers built based on Relu or other activation functions). The convolution kernel size of each convolution layer is 5×5, the number of channels of the output feature map is M or N, and each convolution layer upsamples the width and height by 2 times. It should be noted that the structure of the decoding network model shown in Figure 11 is not used to limit the embodiments of the present application. For example, the convolution kernel size, the number of channels of the feature map, the downsampling multiple, the number of downsampling times, the number of convolution layers, etc. can all be adjusted.

Figure 12 is a schematic diagram of the structure of an entropy estimation network model provided in an embodiment of the present application. The entropy estimation network model includes a super encoding network model (HyEnc), a factor entropy model and a super decoding network model (HyDec). The super encoding network model includes three convolutional layers (Conv) and two activation layers interspersed in cascade (such as activation layers built based on Relu or other activation functions). The convolution kernel size of each convolutional layer is 5×5, the number of channels of the output feature map is M, the first two convolutional layers downsample the width and height by 2 times, and the last convolutional layer does not downsample. The network model structure of the factor entropy model is the same as the network structure of the probability distribution estimation network model introduced above. The super encoding network model includes three convolutional layers (Conv) and two activation layers interspersed in cascade (such as activation layers built based on Relu or other activation functions). The convolution kernel size of each convolutional layer is 5×5, the number of channels of the output feature map is M, the first convolutional layer does not upsample, and the last two convolutional layers upsample the width and height by 2 times. It should be noted that the structure of the entropy estimation network model shown in Figure 12 is not used to limit the embodiments of the present application. For example, the convolution kernel size, the number of channels of the feature map, the downsampling multiple, the number of downsampling times, the upsampling multiple, the number of upsampling times, the number of convolution layers, etc. can all be adjusted.

Fig. 13 is a flow chart of a quantization method of a coding network model provided in an embodiment of the present application. Referring to Fig. 13, the method includes the following steps.

Step 1301: Determine H network units included in the unquantized codec network model, each of the H network units includes a first network layer, a second network layer and a third network layer, the first network layer and the second network layer are both linear operation layers, the third network layer is located between the first network layer and the second network layer, and the third network layer is used to truncate the eigenvalues in the feature map output by the first network layer so that the eigenvalues in the feature map output by the third network layer are all non-negative or all non-positive, and H is an integer greater than or equal to 1.

In some embodiments, multiple third network layers can be determined from an unquantized codec network model, and for any third network layer among the multiple third network layers, if the network layer adjacent to the third network layer is a linear operation layer, the adjacent network layer is used as a first network layer, and if the network layer adjacent to the third network layer is not a linear operation layer, the adjacent network layer is not used as a first network layer. Each third network layer among the multiple third network layers is processed in the same manner, and finally the first network layer among the H network units can be determined.

That is to say, the linear operation layer adjacent to each third network layer in the codec network model can be used as the first network layer. Of course, since not every linear operation layer adjacent to the third network layer in the codec network model will cause abnormal codec results, each linear operation layer included in the codec network model can also be screened to determine the linear operation layer that will cause abnormal codec results, and the linear operation layer that will cause abnormal codec results will be used as the first network layer. Next, the process of determining the linear operation layer that will cause abnormal codec results will be introduced in detail.

Determine M first sample feature maps output by the first linear operation layer, the M first sample feature maps are feature maps corresponding to M normal sample images, M is an integer greater than or equal to 1, the first linear operation layer is any linear operation layer included in the encoding and decoding network model, determine N second sample feature maps output by the first linear operation layer, the N second sample feature maps are feature maps corresponding to N abnormal sample images, N is an integer greater than or equal to 1, and when the M first sample feature maps and the N second sample feature maps meet the feature abnormality condition, determine that the first linear operation layer is the first network layer.

The M feature maps output by the first linear operation layer of the M normal sample images are used as the M first sample feature maps, and the N feature maps output by the first linear operation layer of the N abnormal sample images are used as the N second sample feature maps. Then, based on the M first sample feature maps, the first statistical parameter is determined, based on the N second sample feature maps, the second statistical parameter is determined, and based on the first statistical parameter and the second statistical parameter, it is determined whether the M first sample feature maps and the N second sample feature maps meet the feature abnormality condition.

Optionally, the first linear operation layer includes W channels, the first sample feature map includes feature maps of the W channels, and the first statistical parameter includes first parameters of the W channels. In this case, based on the M first sample feature maps, the implementation process of determining the first statistical parameter includes: counting candidate parameters of the feature map of the target channel included in each of the M first sample feature maps to obtain M candidate parameters, the target channel is any one of the W channels, and based on the M candidate parameters, determining the first parameter of the target channel.

Optionally, the first parameter includes a first maximum value and a first minimum value, the feature map of the target channel in each first sample feature map includes feature values of multiple feature points, and the candidate parameter includes a candidate maximum feature value and a candidate minimum feature value. In this case, the detailed implementation process of determining the first parameter of the target channel includes: counting the maximum feature value and the minimum feature value in the feature map of the target channel included in each first sample feature map to obtain M candidate maximum feature values and M candidate minimum feature values. The first maximum value is determined based on the M candidate maximum feature values, and the first minimum value is determined based on the M candidate minimum feature values.

In some embodiments, the average value of the M candidate maximum eigenvalues can be determined as the first maximum value. In other embodiments, the first maximum value can also be determined based on the M candidate maximum eigenvalues using a sliding average method. In other embodiments, the mean and standard deviation corresponding to the M candidate maximum eigenvalues can also be determined based on the M candidate maximum eigenvalues, and then the sum of three times the standard deviation and the mean can be determined as the first maximum value. In other embodiments, the maximum value among the M candidate maximum eigenvalues can also be determined as the first maximum value. Of course, the first maximum value can also be determined in other ways, and the embodiments of the present application are not limited to this.

The implementation method of determining the first minimum value based on M candidate minimum eigenvalues is similar to the implementation method of determining the first maximum value based on M candidate maximum eigenvalues. For details, please refer to the relevant content above and will not be repeated here.

In some embodiments, the second sample feature map includes feature maps of W channels, and the second statistical parameter includes second parameters of the W channels. In this case, based on the N second sample feature maps, the implementation process of determining the second statistical parameter includes: counting the candidate parameters of the feature map of the target channel included in each of the N second sample feature maps to obtain N candidate parameters, the target channel is any one of the W channels, and based on the N candidate parameters, determining the second parameter of the target channel.

Optionally, the second parameter includes a second maximum value and a second minimum value, the feature map of the target channel in each second sample feature map includes feature values of multiple feature points, and the candidate parameter includes a candidate maximum feature value and a candidate minimum feature value. In this case, the implementation process of determining the second parameter of the target channel is similar to the implementation process of determining the first parameter of the target channel described above. For details, please refer to the relevant content above, which will not be repeated here.

The implementation process of determining whether the M first sample feature graphs and the N second sample feature graphs meet the feature abnormality condition based on the first statistical parameter and the second statistical parameter includes: determining the number of abnormal channels based on the first parameters of the W channels and the second parameters of the W channels, and determining that the M first sample feature graphs and the N second sample feature graphs meet the feature abnormality condition when the number of abnormal channels is greater than or equal to the threshold value of the number of abnormal channels. When the number of abnormal channels is less than the threshold value of the number of abnormal channels, determining that the M first sample feature graphs and the N second sample feature graphs do not meet the feature abnormality condition.

In some embodiments, for any one of the W channels, if the difference between the first maximum value and the second maximum value of the channel is greater than the maximum value difference threshold and the difference between the first minimum value and the second minimum value is greater than the minimum value difference threshold, the channel is determined to be an abnormal channel, and the number of abnormal channels is increased by 1. Otherwise, it is determined that the channel is not an abnormal channel. In the same way, each of the W channels is processed in the above manner to obtain the number of abnormal channels. In other embodiments, for any one of the W channels, if the difference between the first maximum value and the second maximum value of the channel divided by the value of the first maximum value is greater than the maximum value ratio threshold, and the difference between the first minimum value and the second minimum value divided by the value of the first minimum value is greater than the minimum value ratio threshold, the channel is determined to be an abnormal channel, and the number of abnormal channels is increased by 1. Otherwise, it is determined that the channel is not an abnormal channel. In the same way, each of the W channels is processed in the above manner to obtain the number of abnormal channels. Of course, the number of abnormal channels can also be determined in other ways, and the embodiments of the present application are not limited to this.

When the number of abnormal channels is greater than or equal to the threshold value of the number of abnormal channels, it means that the number of abnormal channels is large, and therefore, it can be determined that the M first sample feature graphs and the N second sample feature graphs meet the feature abnormality condition. When the number of abnormal channels is less than the threshold value of the number of abnormal channels, it means that the number of abnormal channels is small, and therefore, it can be determined that the M first sample feature graphs and the N second sample feature graphs do not meet the feature abnormality condition.

When the M first sample feature graphs and the N second sample feature graphs meet the feature abnormality condition, it means that the feature graph output by the first linear operation layer is abnormal, and therefore, it can be determined that the first linear operation layer is the first network layer. When the M first sample feature graphs and the N second sample feature graphs do not meet the feature abnormality condition, it means that the feature graph output by the first linear operation layer is not abnormal, and therefore, it can be determined that the first linear operation layer is not the first network layer.

In some embodiments, the implementation process of determining the second network layer in the H network units included in the unquantized codec network model includes: for the first network layer in each network unit, when there is a linear operation layer after the first network layer, the linear operation layer after the first network layer is determined as the second network layer; when there is no linear operation layer after the first network layer, a linear operation layer is added after the first network layer, and the added linear operation layer is determined as the second network layer.

It should be noted that if the second network layer is an added linear operation layer, the weights included in the second network layer can all be 1 before weight scaling is performed on the second network layer, so that the input feature map and the output feature map of the second network layer are consistent before weight scaling is performed. Optionally, the encoding and decoding network model includes an encoding network model, a decoding network model or an entropy estimation network model.

The abnormal channel number threshold is set in advance and is related to the number of channels in the linear operation layer. For example, the abnormal channel number threshold can be set to 80% of the number of channels in the linear operation layer. The minimum value difference threshold, the maximum value difference threshold, the maximum value ratio threshold and the minimum value ratio threshold are also set in advance and can be adjusted according to different requirements in different situations.

As an example, please refer to Figure 14, which is a schematic diagram of a network unit provided in an embodiment of the present application. In Figure 14, the encoding and decoding network model includes H network units, namely network unit 1, network unit 2 to network unit H.

Step 1302: Scale the weights of each output channel included in the first network layer in each network unit so that the difference between the boundary feature values of each channel in the feature map output by the first network layer in each network unit is less than the feature threshold.

For the first network layer in each network unit, determine K third sample feature maps output by the first network layer, where the K third sample feature maps are feature maps corresponding to K normal sample images, K is an integer greater than or equal to 1, and determine the scaling ratio of each output channel included in the first network layer based on the K third sample feature maps, and scale the weights of each output channel included in the first network layer according to the scaling ratio of each output channel included in the first network layer.

In some embodiments, the reference feature value corresponding to the first network layer and the boundary feature value of each output channel included in the first network layer are determined based on K third sample feature maps, and the ratio between the reference feature value corresponding to the first network layer and the boundary feature value of each output channel included in the first network layer is determined as the scaling ratio of each output channel included in the first network layer.

Based on the K third sample feature graphs, the third statistical parameter of the first network layer is determined, and based on the third statistical parameter of the first network layer, the reference characteristic value corresponding to the first network layer and the boundary characteristic value of each output channel included in the first network layer are determined. Among them, the implementation process of determining the third statistical parameter of the first network layer based on the K third sample feature graphs is consistent with the process of determining the first statistical parameter based on the M first sample feature graphs in the above text. For details, please refer to the relevant content above, which will not be repeated here.

Optionally, the first network layer includes P output channels, the third statistical parameter includes the third parameter of the P output channels, the third parameter may include a third maximum value or a third minimum value, the boundary eigenvalue may be a maximum eigenvalue or a minimum eigenvalue, and for any one of the multiple channels included in the feature map output by the first network layer, the boundary eigenvalue of the channel refers to the maximum eigenvalue or the minimum eigenvalue in the feature map corresponding to the channel in the feature map output by the first network layer. In the case where the meaning of the boundary eigenvalue is different, based on the third statistical parameter of the first network layer, the methods for determining the reference eigenvalue corresponding to the first network layer and the boundary eigenvalues of each output channel included in the first network layer are different. They will be introduced separately below.

When the boundary eigenvalue is the maximum eigenvalue, the third parameter includes a third maximum value, the maximum value of the third maximum values of the P output channels is used as the reference eigenvalue, and the third maximum value of the P output channels is used as the boundary eigenvalue of the P output channels included in the first network layer.

When the boundary eigenvalue is the minimum eigenvalue, the third parameter includes the third minimum value, the minimum value among the third minimum values of the P output channels is used as the reference eigenvalue, and the third minimum value of the P output channels is used as the boundary eigenvalue of the P output channels included in the first network layer.

For any one of the P output channels, the reference eigenvalue corresponding to the first network layer is divided by the value of the boundary eigenvalue of the output channel to determine the ratio between the reference eigenvalue corresponding to the first network layer and the boundary eigenvalue of the output channel. Each of the P output channels is processed in the same manner to obtain the ratio between the reference eigenvalue corresponding to the first network layer and the boundary eigenvalue of each output channel included in the first network layer.

Optionally, according to the scaling ratio of each output channel included in the first network layer, the implementation process of scaling the weights of each output channel included in the first network layer includes: for any one of the P output channels, multiplying the weight of the output channel by the scaling ratio of the output channel to achieve scaling of the weight of the output channel. The weight of each of the P output channels is processed in the same manner to achieve scaling of the weights of each output channel included in the first network layer.

For ease of understanding, the process of scaling the weights of each output channel included in the first network layer is now illustrated by way of example. Referring to Figure 15, the first network layer includes 3 input channels and 5 output channels. The feature map y1=a×x1+b×x2+c×x3 output by output channel 1, and the determination method of y2 to y5 is similar to that of y1, which will not be repeated here. In this case, the weights of output channel 1 are a, b, and c. If the scaling ratio of output channel 1 is 2, then after scaling the weights of output channel 1 according to the scaling ratio of output channel 1, the weights of output channel 1 become 2a, 2b, and 2c.

Please refer to Figure 16. After the first network layer is scaled, the maximum eigenvalues of the feature maps of each channel included in the normal sample feature map and the abnormal sample feature map output by the first network layer are enlarged accordingly, and the maximum eigenvalues of the feature maps of each channel included in the normal sample feature map are consistent.

It should be noted that, when there is a boundary feature value of 0 in a certain output channel among the P output channels included in the first network layer, the weight of the output channel is not scaled. Moreover, for any output channel in any network layer, the output channel can output the feature map corresponding to the channel. If the network layer includes multiple output channels, feature maps corresponding to multiple output channels can be obtained, and the feature maps corresponding to the multiple output channels of the network layer can be referred to as feature maps output by the network layer. In addition, the difference between the boundary feature values of each channel in the feature map output by the first network layer in each of the above network units is less than the feature threshold, which can be understood as follows: for any first network layer, the difference between the boundary feature values of each channel in the feature map output by the first network layer is less than the feature threshold, that is, the difference between the boundary feature values of each channel is less than the feature threshold for the same feature map. Among them, the feature threshold is set in advance, and can also be adjusted according to different needs in different situations.

Step 1303: Scale the weights of each input channel included in the second network layer in each network unit so that the output result of the codec network model is the same before and after the model weight scaling, and the model weight scaling includes weight scaling of each output channel of the first network layer in each network unit and weight scaling of each input channel of the second network layer.

For the second network layer in each network unit, the scaling ratio of each input channel included in the second network layer is determined based on the scaling ratio of each output channel included in the first network layer in the network unit where the second network layer is located, and the weights of each input channel included in the second network layer are scaled according to the scaling ratio of each input channel included in the second network layer.

Optionally, if the first network layer in the network unit where the second network layer is located includes P output channels, the second network layer includes P input channels, and the P input channels of the second network layer correspond one-to-one to the P output channels of the first network layer. For any one of the P input channels, the reciprocal of the scaling ratio of the output channel corresponding to the input channel among the P output channels included in the first network layer in the network unit where the second network layer is located is used as the scaling ratio of the input channel. Each of the P input channels is processed in the same manner to obtain the scaling ratio of each input channel included in the second network layer.

The implementation process of scaling the weights of the input channels included in the second network layer according to the scaling ratio of the input channels included in the second network layer is similar to the implementation process of scaling the weights of the output channels included in the first network layer according to the scaling ratio of the output channels included in the first network layer, and will not be repeated here.

For ease of understanding, the process of scaling the weights of each output channel included in the second network layer is now illustrated by way of example. Referring to Figure 17, the second network layer includes 3 input channels and 5 output channels, and the weights of the input data x1 of input channel 1 on output channels 1 to 5 are a, d, j, g, and m, respectively. The weights of x2 to x5 on output channels 1 to 5 are similar to x1 and will not be repeated here. In this case, the weights of input channel 1 are a, d, j, g, and m. If the scaling ratio of input channel 1 is 0.5, then after scaling the weights of input channel 1 according to the scaling ratio of input channel 1, the weights of input channel 1 become 0.5a, 0.5d, 0.5j, 0.5g, and 0.5m.

Step 1304: quantize the codec network model after the model weights are scaled.

In some embodiments, PTQ can be used to quantize the codec network model after the model weights are scaled. In other embodiments, QAT can also be used to quantize the codec network model after the model weights are scaled. Of course, other methods can also be used to quantize the codec network model after the model weights are scaled, and the embodiments of the present application do not limit this.

Please refer to Figure 18. The reconstructed image 1 in Figure 18 is the reconstructed image obtained by the codec network model after the abnormal sample image is quantized by the codec network model after the model weights are scaled by PTQ. The reconstructed image 2 is the reconstructed image obtained by the codec network model after the abnormal sample image is quantized by QAT after the model weights are scaled. It can be seen from Figure 18 that whether QAT is used to quantize the codec network model after the model weights are scaled, or PTQ is used to quantize the codec network model after the model weights are scaled, the stability of the codec network model can be enhanced, so that the compression rate and the reconstructed image of the image after passing through the codec network model are greatly improved.

To sum up, the quantization method of the codec network model provided in the embodiment of the present application can truncate abnormal eigenvalues greater than the maximum cutoff value when the boundary eigenvalue is the maximum eigenvalue, and can truncate abnormal eigenvalues less than the minimum cutoff value when the boundary eigenvalue is the minimum eigenvalue.

It should be noted that the above-mentioned third network layer has the characteristics of overcoming gradient vanishing and accelerating training speed, and can also increase the nonlinear expression ability of the encoding and decoding network model. In the case where the boundary eigenvalue is the maximum eigenvalue, that is, when the abnormal eigenvalue greater than the maximum truncation value is truncated, the third network layer can be an activation layer constructed based on the Relu function, so that the minimum eigenvalue of the eigenvalue of each feature point in the feature map output by the third network layer is 0, that is, all are non-negative. In this case, since the minimum truncation value is generally less than 0, the abnormal eigenvalue less than the minimum truncation value is truncated to 0 after passing through the third network layer, thereby realizing the correction of the abnormal eigenvalue. In the case where the boundary eigenvalue is the minimum eigenvalue, the third network layer can be an activation layer constructed based on a function after the Relu function is symmetric about the y-axis, so that the maximum eigenvalue of the eigenvalue of each feature point in the feature map output after the third network layer is The value is 0, that is, all are non-positive. In this case, since the maximum cutoff value is generally greater than 0, the abnormal feature value greater than the maximum cutoff value is truncated to 0 after passing through the third network layer, thereby realizing the correction of the abnormal feature value.

As an example, please refer to Figure 19. The first network layer of the six network units included in the codec network model is network layer 1, network layer 2, network layer 3, network layer 4, network layer 5 and network layer 6 in the entropy estimation network model. After the codec network model after model weight scaling is quantized, the process of encoding image 1 is as follows: image 1 passes through the encoding network to obtain feature y. When feature y passes through network layer 1, network layer 2 and network layer 3 in the super-coding network model, the feature map will be corrected according to the cutoff value of each network layer, and finally the prior feature z is obtained. When the prior feature z passes through network layer 4, network layer 5 and network layer 6 in the super-decoding network model in the entropy estimation network model, the feature map will be corrected according to the cutoff value of each network layer, and finally the probability distribution of each feature point of feature y is obtained, and then the feature to be encoded is determined based on the probability distribution of each feature point of feature y, and the feature to be encoded is entropy encoded to obtain a code stream.

By scaling the weights of each output channel included in each first network layer, it is possible to ensure that the difference between the boundary feature values of each channel in the feature map output by each first network layer is less than the feature threshold, and by scaling the weights of the input channels of the second network layer after the first network layer, it is possible to ensure that the output result of the codec network model is the same before and after the model weight scaling. In this way, after the codec network model after the model weight scaling is quantized, the first network layer and the second network layer after the model weight scaling can truncate the abnormal feature values, and can also ensure that the normal feature values are not affected by the model weight scaling, thereby reducing the impact of the codec network model on the weight change of each output channel of the first network layer, so that the final generated code stream and/or reconstructed image does not have abnormalities, and improves the stability of the codec network model.

Since not every linear operation layer in the codec network model will cause abnormal codec results, therefore, in the embodiment of the present application, the linear operation layers included in the codec network model can be screened to determine the linear operation layers that will cause abnormal codec results, and the linear operation layers that will cause abnormal codec results will be used as the first network layer, so that the weights of each output channel included in the linear operation layer that causes abnormal codec results can be targeted and scaled, while reducing the amount of calculation of the codec network model, the stability of the codec network model can also be effectively guaranteed. Moreover, regardless of whether the boundary eigenvalue is the maximum eigenvalue or the minimum eigenvalue, the quantization method of the codec network model provided in the embodiment of the present application can realize the correction of abnormal eigenvalues, so that the final generated code stream and/or reconstructed image does not have abnormalities, thereby improving the stability of the codec network model.

A quantization method for a codec network model provided in an embodiment of the present application can greatly enhance the stability of the quantized codec network model in encoding and decoding various images without increasing the computational intensity of the codec network model and without reducing the performance of the codec network model. It can also be used to enhance the stability of quantization models for other low-level tasks (such as super-resolution tasks, denoising tasks, etc.).

FIG20 is a schematic diagram of the structure of a quantization device for a coding network model provided in an embodiment of the present application, and the quantization device for the coding network model can be implemented by software, hardware, or a combination of both to become part or all of the quantization device for the coding network model. Referring to FIG20 , the device includes: a determination module 2001, a first scaling module 2002, a second scaling module 2003, and a quantization module 2004.

Determination module 2001 is used to determine the H network units included in the unquantized codec network model, each of the H network units includes a first network layer, a second network layer and a third network layer, the first network layer and the second network layer are both linear operation layers, the third network layer is located between the first network layer and the second network layer, and the third network layer is used to truncate the eigenvalues in the feature map output by the first network layer so that the eigenvalues in the feature map output by the third network layer are all non-negative or all non-positive, and H is an integer greater than or equal to 1.

The first scaling module 2002 is used to scale the weights of each output channel included in the first network layer in each network unit so that the difference between the boundary feature values of each channel in the feature map output by the first network layer in each network unit is less than the feature threshold.

The second scaling module 2003 is used to scale the weights of each input channel included in the second network layer in each network unit so that the output result of the codec network model is the same before and after the model weight scaling. The model weight scaling includes weight scaling of each output channel of the first network layer in each network unit and weight scaling of each input channel of the second network layer.

The quantization module 2004 is used to quantize the codec network model after the model weights are scaled.

Optionally, the determination module 2001 is specifically used for:

Determine M first sample feature maps output by the first linear operation layer, where the M first sample feature maps are feature maps corresponding to M normal sample images, M is an integer greater than or equal to 1, and the first linear operation layer is any linear operation layer included in the codec network model;

Determine N second sample feature maps output by the first linear operation layer, where the N second sample feature maps are feature maps corresponding to N abnormal sample images, and N is an integer greater than or equal to 1;

When the M first sample feature maps and the N second sample feature maps meet the feature abnormality condition, the first linear operation layer is determined to be the first network layer.

Optionally, the determination module 2001 is specifically used for:

For the first network layer in each network unit, in the case where there is a linear operation layer after the first network layer, the linear operation layer after the first network layer is determined as the second network layer;

In the case that there is no linear operation layer after the first network layer, a linear operation layer is added after the first network layer, and the added linear operation layer is determined as the second network layer.

Optionally, the first scaling module 2002 is specifically configured to:

For the first network layer in each network unit, determine K third sample feature maps output by the first network layer, where the K third sample feature maps are feature maps corresponding to K normal sample images, and K is an integer greater than or equal to 1;

Determine a scaling ratio of each output channel included in the first network layer based on the K third sample feature maps;

According to the scaling ratio of each output channel included in the first network layer, the weights of each output channel included in the first network layer are scaled.

Optionally, the first scaling module 2002 is specifically configured to:

Determine a reference eigenvalue corresponding to the first network layer and a boundary eigenvalue of each output channel included in the first network layer based on the K third sample feature maps;

The ratio between the reference characteristic value corresponding to the first network layer and the boundary characteristic value of each output channel included in the first network layer is determined as the scaling ratio of each output channel included in the first network layer.

Optionally, the second scaling module 2003 is specifically configured to:

For the second network layer in each network unit, based on the scaling ratios of the output channels included in the first network layer in the network unit where the second network layer is located, determining the scaling ratios of the input channels included in the second network layer;

According to the scaling ratio of each input channel included in the second network layer, the weights of each input channel included in the second network layer are scaled.

Optionally, the encoding and decoding network model includes an encoding network model, a decoding network model or an entropy estimation network model.

Optionally, the boundary eigenvalue is a maximum eigenvalue or a minimum eigenvalue.

By scaling the weights of each output channel included in each first network layer, it is possible to ensure that the difference between the boundary feature values of each channel in the feature map output by each first network layer is less than the feature threshold, and by scaling the weights of the input channels of the second network layer after the first network layer, it is possible to ensure that the output result of the codec network model is the same before and after the model weight scaling. In this way, after the codec network model after the model weight scaling is quantized, the first network layer and the second network layer after the model weight scaling can truncate the abnormal feature values, and can also ensure that the normal feature values are not affected by the model weight scaling, thereby reducing the impact of the codec network model on the weight change of each output channel of the first network layer, so that the final generated code stream and/or reconstructed image does not have abnormalities, and improves the stability of the codec network model.

Since not every linear operation layer in the codec network model will cause abnormal codec results, therefore, in the embodiment of the present application, the linear operation layers included in the codec network model can be screened to determine the linear operation layers that will cause abnormal codec results, and the linear operation layers that will cause abnormal codec results will be used as the first network layer, so that the weights of each output channel included in the linear operation layer that causes abnormal codec results can be targeted and scaled, while reducing the amount of calculation of the codec network model, the stability of the codec network model can also be effectively guaranteed. Moreover, regardless of whether the boundary eigenvalue is the maximum eigenvalue or the minimum eigenvalue, the quantization method of the codec network model provided in the embodiment of the present application can realize the correction of abnormal eigenvalues, so that the final generated code stream and/or reconstructed image does not have abnormalities, thereby improving the stability of the codec network model.

A quantization method for a codec network model provided in an embodiment of the present application can greatly enhance the stability of the quantized codec network model in encoding and decoding various images without increasing the computational intensity of the codec network model and without reducing the performance of the codec network model. It can also be used to enhance the stability of quantization models for other low-level tasks (such as super-resolution tasks, denoising tasks, etc.).

It should be noted that: the quantization device of the codec network model provided in the above embodiment only uses the division of the above functional modules as an example when performing the quantization of the codec network model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the quantization device of the codec network model provided in the above embodiment and the quantization method embodiment of the codec network model belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When the computer program product is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital versatile disc (DVD)) or a semiconductor medium (e.g., a solid state disk (SSD)), etc. It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transitory storage medium.

It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, "/" means or, for example, A/B can mean A or B; "and/or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the feature graphs output by the first network layer and the feature graphs output by the third network layer involved in the embodiments of the present application are obtained with full authorization.

The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims (19)

A quantization method for a coding and decoding network model, characterized in that the method comprises: Determine H network units included in the unquantized codec network model, each of the H network units includes a first network layer, a second network layer and a third network layer, the first network layer and the second network layer are both linear operation layers, the third network layer is located between the first network layer and the second network layer, and the third network layer is used to truncate the feature values in the feature map output by the first network layer, so that the feature values in the feature map output by the third network layer are all non-negative or all non-positive, and H is an integer greater than or equal to 1; Scaling the weights of the respective output channels included in the first network layer in each network unit so that the difference between the boundary feature values of the respective channels in the feature map output by the first network layer in each network unit is less than a feature threshold; Scaling the weights of the respective input channels included in the second network layer in each network unit so that the output result of the encoding and decoding network model is the same before and after the model weight scaling, wherein the model weight scaling includes weight scaling of the respective output channels of the first network layer in each network unit and weight scaling of the respective input channels of the second network layer; The codec network model after the model weights are scaled is quantized. The method of claim 1, wherein determining the first network layer of the H network units included in the unquantized codec network model comprises: Determine M first sample feature maps output by the first linear operation layer, where the M first sample feature maps are feature maps corresponding to M normal sample images, M is an integer greater than or equal to 1, and the first linear operation layer is any linear operation layer included in the encoding and decoding network model; Determine N second sample feature maps output by the first linear operation layer, where the N second sample feature maps are feature maps corresponding to N abnormal sample images, and N is an integer greater than or equal to 1; When the M first sample feature maps and the N second sample feature maps meet the feature abnormality condition, the first linear operation layer is determined to be the first network layer. The method according to claim 1 or 2, characterized in that determining the second network layer in the H network units included in the unquantized codec network model comprises: For the first network layer in each network unit, in the case where there is a linear operation layer after the first network layer, determining the linear operation layer after the first network layer as the second network layer; In the case that there is no linear operation layer after the first network layer, a linear operation layer is added after the first network layer, and the added linear operation layer is determined as the second network layer. The method according to any one of claims 1 to 3, characterized in that scaling the weights of each output channel included in the first network layer in each network unit comprises: For the first network layer in each network unit, determine K third sample feature maps output by the first network layer, where the K third sample feature maps are feature maps corresponding to K normal sample images, and K is an integer greater than or equal to 1; Determine a scaling ratio of each output channel included in the first network layer based on the K third sample feature maps; The weights of the output channels of the first network layer are scaled according to the scaling ratios of the output channels of the first network layer. The method of claim 4, wherein determining the scaling ratio of each output channel included in the first network layer based on the K third sample feature maps comprises: Determine, based on the K third sample feature maps, a reference feature value corresponding to the first network layer and a boundary feature value of each output channel included in the first network layer; The ratio between the reference characteristic value corresponding to the first network layer and the boundary characteristic value of each output channel included in the first network layer is determined as the scaling ratio of each output channel included in the first network layer. The method according to any one of claims 1 to 5, characterized in that scaling the weights of each input channel included in the second network layer in each network unit comprises: For the second network layer in each network unit, determining a scaling ratio of each input channel included in the second network layer based on a scaling ratio of each output channel included in the first network layer in the network unit where the second network layer is located; The weights of the input channels included in the second network layer are scaled according to the scaling ratios of the input channels included in the second network layer. The method according to any one of claims 1 to 6 is characterized in that the encoding and decoding network model includes an encoding network model, a decoding network model or an entropy estimation network model. The method according to any one of claims 1 to 7, characterized in that the boundary eigenvalue is a maximum eigenvalue or a minimum eigenvalue. A quantization device for a coding and decoding network model, characterized in that the device comprises: A determination module, used to determine H network units included in the unquantized codec network model, each of the H network units includes a first network layer, a second network layer and a third network layer, the first network layer and the second network layer are both linear operation layers, the third network layer is located between the first network layer and the second network layer, and the third network layer is used to truncate the feature values in the feature map output by the first network layer so that the feature values in the feature map output by the third network layer are all non-negative or all non-positive, and H is an integer greater than or equal to 1; A first scaling module is used to scale the weights of each output channel included in the first network layer in each network unit so that the difference between the boundary feature values of each channel in the feature map output by the first network layer in each network unit is less than a feature threshold; A second scaling module is used to scale the weights of each input channel included in the second network layer in each network unit so that the output result of the encoding and decoding network model is the same before and after the model weight scaling, and the model weight scaling includes weight scaling of each output channel of the first network layer in each network unit and weight scaling of each input channel of the second network layer; A quantization module is used to quantize the encoding and decoding network model after the model weights are scaled. The device according to claim 9, wherein the determining module is specifically configured to: Determine M first sample feature maps output by the first linear operation layer, where the M first sample feature maps are feature maps corresponding to M normal sample images, M is an integer greater than or equal to 1, and the first linear operation layer is any linear operation layer included in the encoding and decoding network model; Determine N second sample feature maps output by the first linear operation layer, where the N second sample feature maps are feature maps corresponding to N abnormal sample images, and N is an integer greater than or equal to 1; When the M first sample feature maps and the N second sample feature maps meet the feature abnormality condition, the first linear operation layer is determined to be the first network layer. The device according to claim 9 or 10, characterized in that the determination module is specifically used to: For the first network layer in each network unit, in the case where there is a linear operation layer after the first network layer, determining the linear operation layer after the first network layer as the second network layer; In the case that there is no linear operation layer after the first network layer, a linear operation layer is added after the first network layer, and the added linear operation layer is determined as the second network layer. The device according to any one of claims 9 to 11, wherein the first scaling module is specifically configured to: For the first network layer in each network unit, determine K third sample feature maps output by the first network layer, where the K third sample feature maps are feature maps corresponding to K normal sample images, and K is an integer greater than or equal to 1; Determine a scaling ratio of each output channel included in the first network layer based on the K third sample feature maps; The weights of the output channels of the first network layer are scaled according to the scaling ratios of the output channels of the first network layer. The device according to claim 12, wherein the first scaling module is specifically configured to: Determine, based on the K third sample feature maps, a reference feature value corresponding to the first network layer and a boundary feature value of each output channel included in the first network layer; The ratio between the reference characteristic value corresponding to the first network layer and the boundary characteristic value of each output channel included in the first network layer is determined as the scaling ratio of each output channel included in the first network layer. The device according to any one of claims 9 to 13, wherein the second scaling module is specifically configured to: For the second network layer in each network unit, determining a scaling ratio of each input channel included in the second network layer based on a scaling ratio of each output channel included in the first network layer in the network unit where the second network layer is located; The weights of the input channels included in the second network layer are scaled according to the scaling ratios of the input channels included in the second network layer. The device as described in any one of claims 9 to 14 is characterized in that the encoding and decoding network model includes an encoding network model, a decoding network model or an entropy estimation network model. The device as described in any one of claims 9 to 15 is characterized in that the boundary eigenvalue is a maximum eigenvalue or a minimum eigenvalue. A quantization device for a codec network model, the quantization device for the codec network model comprising a processor and a memory, the memory being used to store a program for executing any of the methods described in claims 1-8, and the processor being configured to execute the program stored in the memory. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented. A computer program product, characterized in that the computer program product comprises computer instructions, and the computer instructions implement any one of the methods of claims 1-8 when executed on a computer.