WO2022021083A1 - Image processing method, image processing device, and computer readable storage medium - Google Patents

Abstract

Disclosed in embodiments of the present application is an image processing method, applied to a neural network. The method comprises: determining a target dynamic range of an image to be processed, said image being used for being inputted into the neural network; determining a target quantization parameter set adapted to the target dynamic range from multiple candidate quantization parameter sets, different quantization parameter sets being adapted to different dynamic ranges; performing quantization processing on said image according to the target quantization parameter set, such that image data of said image is quantified into fixed-point data whose accuracy is adapted to the target dynamic range. The image processing method disclosed in the embodiments of the present application solves the technical problem that the accuracy of an output result of the neural network is insufficient after quantization is performed by existing quantization methods.

Description

Image processing method, image processing device, and computer-readable storage medium technical field

The present application relates to the technical field of image processing, and in particular, to an image processing method, an image processing apparatus, and a computer-readable storage medium.

Background technique

Neural networks have achieved great success in image processing, speech recognition and processing, and are widely used in embedded devices. Since the neural network often contains many parameters, the entire operation process needs to consume a lot of storage and computing resources, which makes it difficult to meet the requirements of high real-time performance and low power consumption when running the neural network in low-cost embedded devices. Require.

Quantizing the neural network is a method to improve the operation speed and reduce the energy consumption. However, after the neural network is quantized by the existing quantization method, the accuracy of the output result of the neural network is insufficient.

SUMMARY OF THE INVENTION

In view of this, one of the purposes of the present application is to solve the technical problem of insufficient precision of the output result of the neural network after quantization by the existing quantization method.

A first aspect of the embodiments of the present application provides an image processing method, which is applied to a neural network, and the method includes:

determining the target dynamic range of the image to be processed, the image to be processed is used to input the neural network;

determining a target quantization parameter set adapted to the target dynamic range from a plurality of candidate quantization parameter sets, wherein different said quantization parameter sets are adapted to different dynamic ranges;

According to the target quantization parameter set, quantization processing is performed on the to-be-processed image, and the image data of the to-be-processed image is quantized into fixed-point data whose precision is adapted to the target dynamic range.

A second aspect of the embodiments of the present application provides an image processing apparatus applied to a neural network, including: a processor and a memory storing a computer program;

The processor implements the following steps when executing the computer program:

determining the target dynamic range of the image to be processed, the image to be processed is used to input the neural network;

Determine a target quantization parameter set adapted to the target dynamic range from a plurality of candidate quantization parameter sets, wherein different described quantization parameter sets are adapted to different dynamic ranges;

According to the target quantization parameter set, quantization processing is performed on the to-be-processed image, and the image data of the to-be-processed image is quantized into fixed-point data whose precision is adapted to the target dynamic range.

A third aspect of the embodiments of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium; when the computer program is executed by a processor, any image processing method of the first aspect above is implemented.

In the image processing method provided by the embodiment of the present application, different quantization parameters are set for input images with different dynamic ranges, so that, for the image to be processed, the quantization parameter suitable for the dynamic range of the image to be processed can be used for the image to be processed. The image data of the processed image is quantized, so that the fixed-point image data obtained after quantization can have sufficient accuracy, thereby reducing the accuracy loss caused by quantization, and can meet the high-precision image regression tasks such as image denoising and super-resolution. Require.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present application.

FIG. 2 is a system block diagram of an early stage of an image processing method provided by an embodiment of the present application.

FIG. 3 is a system block diagram of an application stage of an image processing method provided by an embodiment of the present application.

FIG. 4 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Deep learning techniques have achieved great success in image processing, speech recognition and processing, and are widely used in embedded devices. Neural network is a deep learning algorithm. Because neural network often contains many parameters, the whole operation process needs to consume a lot of storage and computing resources, which makes it difficult to meet high real-time requirements when implemented in low-cost embedded devices. performance and low power requirements. In order to reduce the computational complexity of the neural network and improve the computational speed, quantizing the neural network is a mainstream method.

Quantization approximates data from one range of values to another, for example, floating-point data can be quantized to fixed-point data. In computers, data can be represented in two forms: floating-point numbers and fixed-point numbers. Among them, a fixed-point number can be understood as a number with fixed decimal places, such as a 16-bit binary data, 5 bits can be fixedly used to store decimals, and the remaining 11 bits can be used to store integers. The floating-point number can be understood as a number whose decimal places are not fixed and can be floated according to requirements.

Since floating-point numbers require more storage space than fixed-point numbers, and most processors are currently better at fixed-point operations, after quantizing the floating-point data in the neural network into fixed-point data, the operation speed of the neural network can be improved. , the occupied storage resources can be reduced, and the required energy consumption can also be reduced.

However, although quantizing the neural network can have the above-mentioned series of effects, the applicant has found that after the existing quantization methods quantify the neural network, the output results of the neural network are difficult to achieve the required accuracy, especially for denoising, super Image regression tasks such as resolution, image regression tasks require higher accuracy than image classification tasks.

Specifically, as can be seen from the foregoing, quantization can realize the conversion of floating-point numbers to fixed-point numbers, and the conversion of floating-point numbers to fixed-point numbers can be regarded as a process of determining the number of decimal places. For a data, the number of decimal places determines the precision of the data, and the number of integer bits determines the numerical value that the data can express. Since the total number of bits of data is usually fixed, such as 4, 8, 16, or 32 bits, etc., the more decimal places, the fewer integer digits. Therefore, the ideal representation of a data is that there are just enough integer digits to express its value, and the remaining digits are used as decimal places to express higher precision.

However, in the related quantization method for neural networks, since it considers universality more, the determined quantization parameters are more suitable for input images with a large dynamic range. The quantization parameter can determine the number of decimal digits and integer digits of the data obtained after quantization. For the quantization parameter suitable for the input image with a large dynamic range, the image data obtained after quantization will have more integer digits to ensure that it is sufficient to express The actual pixel value of the image is obtained, but the number of decimal places is correspondingly smaller. For an input image with a small dynamic range, the representation of the pixel value does not require so many integer bits, but due to the use of the same set of quantization parameters, the image data will have excess integer bits and not exhausted decimal bits. The accuracy of the data is insufficient, and ultimately the accuracy of the output of the neural network is also insufficient.

Based on this, an embodiment of the present application provides an image processing method. Referring to FIG. 1 , FIG. 1 is a flowchart of the image processing method provided by the embodiment of the present application. The image processing method can be applied to a neural network, such as a convolutional neural network, and the method includes the following steps:

S101. Determine the target dynamic range of the image to be processed.

The image to be processed can be used to feed the neural network. It should be noted that the neural network is a pre-trained neural network, and the specific training process may refer to the prior art, which will not be repeated here. Moreover, the function corresponding to the neural network may be any function, such as image denoising, face recognition, super-resolution, scene recognition, etc., which is not limited here.

The dynamic range of an image may be the range of pixel values of the image. For example, for an image whose maximum pixel value is 60 and the minimum pixel value is 0, its dynamic range may be expressed as 0-60.

S102. Determine a target quantization parameter set adapted to the above-mentioned target dynamic range from a plurality of candidate quantization parameter sets.

S103. Perform quantization processing on the image to be processed according to the target quantization parameter set, and quantize the image data of the image to be processed into fixed-point data whose precision is adapted to the target dynamic range.

Among them, different quantization parameter sets can be adapted to different dynamic ranges. These candidate quantization parameter sets may be predetermined, and the specific determination method will be described later. For an image, when the image is quantized according to the quantization parameter set suitable for the dynamic range of the image, the image data of the image can be quantized into fixed-point data whose precision is suitable for the dynamic range of the image.

The so-called adaptation of precision and dynamic range can be understood in conjunction with the ideal representation of the data described above. For example, the integer number of bits of the fixed-point data can be just enough to express the pixel value of the image. For example, the total number of bits is 8 bits. The maximum pixel value of the image is 15, then when the number of integer digits is 4, it is just enough to express the pixel value of the image, and the remaining 4 digits are decimal digits. At this time, the data precision of the image is the highest. Of course, in the above example, if it is considered that the precision of more than 2 decimal digits of the image data meets the requirements, the integer digits of the fixed-point data can also be 5 or 6 digits. At this time, although the integer digits are slightly more than With just enough bits to express the pixel values of the image, the precision of the fixed-point data can still be considered to fit the dynamic range of the image.

After the image to be processed is quantized, the image data of the image to be processed is processed as precision-adapted (for convenience, precision and dynamic range adaptation is simply referred to as precision-adapted) fixed-point data, the fixed-point image data of the to-be-processed image Can be used to input neural networks.

In the image processing method provided by the embodiment of the present application, different quantization parameters are set for input images with different dynamic ranges, so that, for the image to be processed, the quantization parameter suitable for the dynamic range of the image to be processed can be used for the image to be processed. The image data of the processed image is quantized, so that the fixed-point image data obtained after quantization can have sufficient accuracy on the basis of no distortion (that is, the value is consistent with the actual size), thereby reducing the accuracy loss caused by quantization. It meets the high-precision requirements of image regression tasks such as image denoising and super-resolution.

In one embodiment, the image to be processed may be an image block in the original image. Specifically, the original image may be the processing object of the image processing task, and the original image may be processed in blocks, thereby obtaining multiple image blocks of the original image, and each image block may be the to-be-to-be described in the embodiments of the present application. Process images. For each image block, the image processing method provided in the embodiment of the present application can be used to perform quantization and input neural network for forward propagation and other processing, so as to obtain an output image corresponding to each image block output by the neural network. The output images corresponding to each image block can be fused to obtain a complete target output image, which is the output result corresponding to the original image.

In the above embodiment, since the original image is divided into blocks, different processing can be performed for image blocks with different dynamic ranges, and the obtained output image corresponding to each image block has high precision, and the target output obtained by final fusion The image will have a larger accuracy improvement.

When the original image is divided into blocks, there can be various implementations. For example, the original image can be divided evenly, so that the size of each divided image block is consistent, and for example, it can be divided unevenly, so that the image blocks can be divided The sizes may be different. For another example, there may be overlapping divisions, so that there are overlapping parts between image blocks... This embodiment of the present application does not specifically limit the way of dividing the original image.

It should be noted that if the original image is divided with overlapping, when each image block is used for fusion, the overlapping area between the image blocks can be deduplicated, so that one copy of the image data in the same area can be reserved. , and then use the de-duplicated image blocks to fuse to obtain the target output image.

In one embodiment, the original network parameters of the trained neural network may be floating-point network parameters. Since the quantized image to be processed input to the neural network is fixed-point data, before the quantized image to be processed is input, The original network parameters of the neural network are converted into fixed-point network parameters, so that the fixed-point image data of the image to be processed and the fixed-point network parameters of the neural network can carry out forward propagation of the neural network according to the algorithm of fixed-point numbers.

The network parameters of the neural network may include network parameters of each layer of the neural network, and these network parameters may specifically include one or more of weight parameters, bias parameters, and the like. Of course, different neural networks can have different network parameters. For example, a neural network can include one or more layers such as convolutional layers, fully connected layers, and nonlinear activation layers. The network parameters corresponding to each layer can be different. The application examples are not limited herein.

Regarding the specific implementation of converting the original network parameters of the neural network into fixed-point network parameters, an implementation manner is that the original network parameters of the neural network can be quantized to obtain fixed-point network parameters corresponding to the original network parameters. There are many quantization methods, such as linear mapping, shifting, truncation, etc. For ease of understanding, the quantization method of uniform linear mapping may be used as an example for description.

Suppose there is a floating-point weight parameter whose value range is [X _min ,X _max ], if the floating-point weight parameter needs to be quantized to 8bits representation, that is, the floating-point weight parameter is quantized to the value range of [0,255] , the floating-point number can be converted to a fixed-point number by two quantization parameters s and z. The quantification formula is as follows:

y=clamp(round(s*x+z), 0, 255)

Among them, x is the floating-point value before quantization, y is the fixed-point value after quantization, s and z are both quantization parameters, where s is the quantization step size, z is zero, and round() is the rounding function. After the quantization parameters s and z are determined, the floating-point number can be quantized into a fixed-point number through the above formula.

It should be noted that the quantization parameters corresponding to different quantization methods may be different. In the above example, the quantization parameters corresponding to the uniform linear mapping include the quantization step size and the zero point, but the quantization parameters corresponding to other quantization methods are not necessarily the same as the uniform linear mapping, that is, the quantization step size s and the zero point z are not necessarily the same.

After the quantization object (that is, some kind of data to be quantized) is determined, there are various specific determinations to determine the quantization parameter corresponding to the quantization object. In one embodiment, the quantization parameter may be determined by minimizing the KL divergence between the pre-quantization data distribution and the post-quantization data distribution. In another embodiment, the quantization parameter can also be determined by counting the actual maximum and minimum values.

In one embodiment, different quantization parameter sets may correspond to different dynamic range intervals, for example, the first quantization parameter set may correspond to the interval [0, 63], and the second quantization parameter set may correspond to the interval [0, 127]... Of course, the above The interval is only an example, and a specific interval corresponding to a quantization parameter set can be set by those skilled in the art according to requirements, which is not limited here. Correspondingly, when determining the target quantization parameter set applicable to the image to be processed, the minimum interval to which it belongs can be calculated according to the target dynamic range of the image to be processed, and then the quantization parameter set corresponding to the minimum interval can be determined as the target quantization. parameter set.

It should be noted that there may be a certain degree of flexibility when determining the minimum interval to which the target dynamic range belongs. For example, the dynamic range of the image to be processed is 0-64. If the dynamic range interval is [0,63] and [0,127], in one embodiment, it can be directly determined that the minimum interval to which the target dynamic range belongs is [0,127]. In another implementation manner, considering that the quantization parameter set corresponding to the [0, 127] interval is lower in accuracy, and the dynamic range of 0-64 does not exceed [0, 63] by much, therefore, 0 If the dynamic range of -64 is reduced to a certain extent, the minimum interval to which the reduced target dynamic range belongs may be the interval of [0,63]. Of course, in another example, the target dynamic range may also be amplified to an appropriate degree, and then the minimum interval to which the amplified target dynamic range belongs may be determined.

The image to be processed is the input image of the input layer of the neural network. After the image to be processed is quantized, the fixed-point image data of the image to be processed can be input to the neural network for forward propagation. In one embodiment, quantization processing may also be performed on the input images of other layers of the neural network in the process of forward propagation. In the neural network, except for the input layer, the input images of other layers can be the output images of the previous layer, and the operation performed by each layer can be considered as feature extraction for the input image. Taking the convolutional neural network as an example, The input image of the Nth layer is convolved with the convolution kernel of the Nth layer to obtain a feature map, which can be used as the input image of the N+1 layer.

In this embodiment, one quantization parameter set may include multiple quantization parameter subsets, and each quantization parameter subset may correspond to a layer of the neural network. For example, if the neural network has N layers, there can be N quantization parameter subsets, and the quantization parameter subset corresponding to each layer can be used to quantize the input image of the layer. For example, the quantization parameter subset of the 0th layer can be It is used to quantize the input image (ie, the image to be processed) of the 0th layer (ie, the input layer), and the quantization parameter subset of the second layer can be used to quantize the input image of the second layer.  … Nth After the input image of the layer is quantized by the quantization parameter subset corresponding to the Nth layer, the quantized input image of the Nth layer can be input to the Nth layer, and perform operations such as convolution with the network parameters of the Nth layer.

The quantization method described in the above embodiments can also be described as layer-by-layer quantization. For a convolutional neural network, that is, the convolution kernels of the same layer correspond to the same set of quantization parameters.

In one embodiment, in order to further improve the accuracy, a channel-by-channel quantization method can also be used, that is, on the basis of layer-by-layer quantization, each convolution kernel in each layer of the neural network can be further quantized separately. Specifically, if the first convolution kernel is any convolution kernel in the neural network, the first convolution kernel may be quantized according to the quantization parameter corresponding to the first convolution kernel.

Taking the Nth layer as an example, if the input image of the Nth layer is 32*32*16, 32*32 represents the size of the input image, 16 represents the number of channels of the input image, and the Nth layer includes 32 convolution kernels , each convolution kernel can be 5*5*16, 5*5 indicates the size of the convolution kernel, and 16 indicates the number of channels of the convolution kernel, then each convolution kernel of the 32 convolution kernels can be The kernel determines the quantization parameters respectively, and can perform quantization processing on each of the 32 convolution kernels according to the determined quantization parameters. In this way, after the input image of the Nth layer is quantized by the quantization parameter subset of the Nth layer, the quantized input image of the Nth layer can be respectively processed with each quantized convolution kernel in the Nth layer. Convolution operation.

It can be seen from the foregoing that the candidate quantization parameter set may be predetermined, and the stage in which the quantization parameter set is predetermined may be referred to as an early stage. In an implementation manner, multiple dynamic range intervals may be preset, and relevant descriptions have been made in the previous section, such as [0, 63], [0, 127]. For each interval, its corresponding quantization parameter set can be determined. Taking the first interval as an example, the first interval can be any interval in multiple dynamic range intervals, and a plurality of sample images whose dynamic ranges belong to the first interval can be obtained, and the quantization corresponding to the first interval can be determined according to the sample images. parameter set.

When the sample image is acquired, an implementation manner is that the sample image can be obtained by dividing the material image into blocks. After the dynamic range is divided into intervals, the dynamic range corresponding to some intervals may be small. In order to obtain the sample images whose dynamic range belongs to the interval, the sample images of the corresponding interval can be more easily obtained by dividing the material image into blocks.

If the quantization parameter set includes multiple quantization parameter subsets, and each quantization parameter subset corresponds to a layer of the neural network, when determining the quantization parameter set corresponding to the first interval, the above dynamic range may belong to the first interval. The sample images are respectively input to the neural network for forward propagation, and the input images of each layer of the neural network are obtained. For the input image of each layer, please refer to the relevant description above. The input image of the input layer is the input sample image, and the input image of the Nth layer is the output image of the N-1th layer.

After the input image of each layer of the neural network is obtained, a quantization parameter (quantization parameter subset) for quantizing the image data of the input image of the layer can be determined for each layer, and the determined quantization parameter can be The image data of the input image is quantized into fixed-point data with a precision that matches the dynamic range of the input image. The specific way of determining the quantization parameters has been described above, such as minimizing the KL divergence between the pre-quantization data distribution and the post-quantization data distribution, counting the actual maximum and minimum values, and so on.

It should be noted that, in the early stage of determining the quantization parameter set, the network parameters of the neural network may be floating-point network parameters, and the sample images used to input the neural network may also be floating-point image data. In one embodiment, the network parameters of the neural network can directly use their original floating-point network parameters. In another implementation manner, the original network parameters of the neural network may be quantized first to obtain fixed-point network parameters, and the specific implementation of quantization to obtain fixed-point network parameters has been described above, and will not be repeated here. Furthermore, the fixed-point network parameters can be inversely quantized to obtain floating-point network parameters used for operations such as convolution with the sample image.

The specific implementation of inverse quantization can be obtained by inverse derivation according to the method used for quantization. For example, the method used for quantization is uniform linear mapping, then inverse quantization can be achieved by the following formula:

x'=(y-z)/s

Among them, y is a fixed-point value after quantization, x' is a floating-point value after inverse quantization, and s and z are quantization parameters.

Since in the application stage, the network parameters used by the neural network of the image input to be processed are fixed-point network parameters, so in the early stage, in order to fit with the application stage to make the determined quantitative parameter set more applicable, the network parameters of the neural network can be Instead of using the original network parameters, the floating-point network parameters obtained by inverse quantization of the fixed-point network parameters described above are used.

In one embodiment, the set of quantization parameters may also be determined in real-time during the forward propagation of the neural network. Specifically, the quantization parameter subset of the 0th layer can be determined according to the dynamic range of the image to be processed, and according to the determined quantization parameter subset of the 0th layer, the quantization process can be performed on the to-be-processed image, and the quantization obtained by the to-be-processed The fixed-point image data corresponding to the image can be input to the 0th layer (input layer) to obtain the output image of the 0th layer, that is, the input image of the 1st layer. The quantization parameter subset of the first layer can be determined according to the dynamic range of the input image of the first layer, and the input image of the first layer can be quantized according to the determined quantization parameter subset of the first layer, and the obtained quantization parameter The latter input image of the first layer can be input to the first layer to obtain the output image of the first layer, that is, the input image of the second layer. Then the quantization parameter subset of the second layer can be determined according to the dynamic range of the input image of the second layer...

The above method of determining the quantization parameter set in real time during the forward propagation of the neural network needs to consume more storage and computing resources because the output image of each layer needs to be cached to determine the quantization parameter according to its dynamic range. In contrast, pre-determining candidate quantization parameter sets in the early stage can occupy less storage and computing resources in the application stage.

As described above, in the embodiments of the present application, the neural network inputted by the image is a neural network that has been trained and has a specific function. In the training phase of the neural network, the image data used for inputting the neural network for training may be preprocessed. For example, the image data used for training may be subjected to zero-mean processing and/or prior to input to the neural network. Normalization processing to speed up the convergence of the network parameters of the neural network during training. Therefore, for a trained neural network, before the image data of images such as images to be processed or sample images are input, these image data can also be preprocessed. In one example, the pixel values of each channel of the input image can be preprocessed by the following formula:

y _i =k*x _i +b

Among them, x represents the input image, i represents the input image channel, for RGB color images, i=1, 2, 3; y represents the output image, the parameter k represents the scaling factor of the pixel value, and the parameter b represents the bias of the pixel value. Shift coefficient, parameters k and b are floating point numbers.

A relatively detailed example is provided below.

This embodiment may include a pre-stage and an application stage.

For the early stage, reference may be made to Fig. 2, which is a system block diagram of the early stage of an image processing method provided by an embodiment of the present application.

In the early stage, a plurality of material images for inputting the convolutional neural network for forward propagation may be obtained, and preprocessing such as zero mean and/or normalization may be performed on the material images. The preprocessed material image can be processed into blocks to obtain multiple image blocks, each image block can be used as a sample image.

The dynamic range of each sample image can be counted, and multiple dynamic range intervals can be defined according to the statistical results. For example, after the dynamic range of sample images is counted, if it is determined that the dynamic range of each sample image is distributed in the interval [-128, 127] (due to the sample image is preprocessed, its pixel value can be negative), then it can be obtained from [ -128,127] define multiple intervals, for example, you can define six intervals [-64,63], [-64,0), [0,63], [-128,127], [-128,0) and [0,127] .

For the original network parameters of the convolutional neural network, quantization can be performed first to obtain fixed-point network parameters, and then by inverse quantization of the fixed-point network parameters, floating-point network parameters can be obtained, and the floating-point network parameters can be applied to the convolutional neural network. .

Each sample image is input into the convolutional neural network for forward propagation, and the output results of each layer of the convolutional neural network are obtained. Since the output of the Nth layer is also the input of the N+1th layer, obtaining the output result of each layer can also be regarded as obtaining the image data of the input image of each layer (the input image of the input layer is the sample image itself, other The input image of the layer can be a feature map obtained by performing feature extraction on the sample image).

Quantization can choose layer-by-layer quantization or channel-by-channel quantization. Layer-by-layer quantization means that each layer corresponds to a set of quantization parameters, and channel-by-channel quantization means that each convolution kernel corresponds to a set of quantization parameters. The following takes layer-by-layer quantization as an example for description.

For each sample image, the minimum interval to which it belongs can be determined, and when the minimum interval is determined, the dynamic range of the sample image can be appropriately scaled. The image data of the input image of each layer obtained after the sample image is input to the convolutional neural network can be classified according to the interval to which the sample image belongs, and further, for each interval, the image data of the input image of each layer can be determined. The corresponding quantization parameter is obtained, and the quantization parameter subset corresponding to the layer in this interval is obtained.

For example, for example, if there are 5 sample images whose dynamic range belongs to the [-64,63] interval, after the 5 sample images are respectively input into the convolutional neural network, the image that is the input image of the Nth layer is obtained. data, for the image data of the input image that is also the Nth layer, determine the quantization parameter subset corresponding to the Nth layer in the [-64,63] interval. The quantization parameter subset corresponding to each layer in the [-64,63] interval may constitute the quantization parameter set corresponding to the [-64,63] interval.

For the application stage, reference may be made to FIG. 3 , which is a system block diagram of the application stage of an image processing method provided by an embodiment of the present application.

In the early stage, the quantization parameter sets corresponding to different dynamic range intervals have been determined. In the application stage, if the target object of the image processing task is the original image, image preprocessing can be performed on the original image first. The preprocessed original image can be processed into blocks, and each image block can be called the image to be processed. The following describes each image block, that is, the image to be processed.

For the image to be processed, its dynamic range (target dynamic range) can be determined first. According to the minimum interval to which the target dynamic range belongs, the quantization parameter set corresponding to the minimum interval is determined as the target quantization parameter set.

The target quantization parameter set can be applied to the convolutional neural network, and the fixed-point network parameters obtained by quantizing the original network parameters in the previous stage can be applied to the convolutional neural network. The target quantization parameter set includes the quantization parameter subset corresponding to each layer, then in the 0th layer (ie the input layer), the image to be processed can be quantized according to the quantization parameter subset corresponding to the 0th layer. The fixed-point image data of the image to be processed can be input to layer 0. The output of the 0th layer is the image data of the input image of the first layer. The input image of the first layer can be quantized according to the quantization parameter subset corresponding to the first layer, and the quantized input image of the first layer can be processed. You can input the first layer and perform operations such as convolution with the fixed-point network parameters of the first layer to obtain the output of the first layer, that is, the image data of the input image of the second layer... The processing flow of other subsequent layers is the same, here Not one by one. In the convolutional neural network, the fixed-point image data and the fixed-point network parameters perform convolution and other operations layer by layer according to the fixed-point number algorithm, and finally the convolutional neural network can output the output image corresponding to the image to be processed.

Since the image to be processed is only one image block of the original image, the output images corresponding to each image block can be further fused to obtain the final target output image.

In an embodiment, the image to be processed can also be obtained by the following manner: the region of interest in the original image can be determined, the region of interest can be segmented or cut out, and the obtained image block can be used as the image to be processed. It is understandable that the area of interest can be the area with high attention of the human eye. Since the human eye will give more attention to these areas, there are higher requirements for processing details. If the processing accuracy is insufficient, it will be easily detected. The human eye detects flaws.

The region of interest can be determined according to the content of the image, for example, it can be the subject of a person, an animal, etc. in the image, or it can be a specific area of the person in the image, such as the face, hair, and clothing. When determining the region of interest in the original image, in one embodiment, the system may determine it by itself through an algorithm. Specifically, the original image may be semantically segmented to determine the object category contained in the original image. Object categories can be the aforementioned characters, animals, or parts of characters, such as hair, faces, and other different categories. For each object category, its corresponding attention degree can be determined. The determination of the degree of attention can be determined according to the preset correspondence between the object category and the degree of attention. Of course, the degree of attention can also be calculated by formulas and other methods. After the attention degree is calculated, the area with the highest attention degree can be regarded as the area of interest.

Of course, in an implementation manner, the region of interest may also be an area selected by the user. For example, during specific interaction, the user may be provided with information about each object in the identified image, and the user may select the object of interest, Various selection tools can also be provided to the user, and the user can define the region of interest by himself.

By processing the region of interest by using the image processing method provided in the embodiment of the present application, the processing of the region of interest can have higher precision, and the region of interest in the output image can have more delicate effects, such as presenting more details. Moreover, only processing the region of interest can achieve the effect of saving computing power compared to processing the entire image.

The above is a detailed description of the image processing method provided by the embodiments of the present application. In the image processing method provided by the embodiment of the present application, different quantization parameters are set for input images with different dynamic ranges, so that, for the image to be processed, the quantization parameter suitable for the dynamic range of the image to be processed can be used for the image to be processed. The image data of the processed image is quantized, so that the fixed-point image data obtained after quantization can have sufficient accuracy on the basis of no distortion (that is, the value is consistent with the actual size), thereby reducing the accuracy loss caused by quantization, so that the The output of the neural network has higher accuracy, which can meet the high-precision requirements of image regression tasks such as image denoising and super-resolution.

Referring to FIG. 4 below, FIG. 4 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. The image processing apparatus can be applied to a neural network, including: a processor 410 and a memory 420 storing a computer program;

The processor implements the following steps when executing the computer program:

determining the target dynamic range of the image to be processed, the image to be processed is used to input the neural network;

determining a target quantization parameter set adapted to the target dynamic range from a plurality of candidate quantization parameter sets, wherein different said quantization parameter sets are adapted to different dynamic ranges;

According to the target quantization parameter set, quantization processing is performed on the to-be-processed image, and the image data of the to-be-processed image is quantized into fixed-point data whose precision is adapted to the target dynamic range.

Optionally, the to-be-processed image is an image block in the original image;

The processor is further configured to perform fusion processing on the output images corresponding to each image block in the original image to obtain a target output image corresponding to the original image, wherein the output image takes the image block as an input the image output by the neural network.

Optionally, when the processor performs fusion processing on the output image corresponding to each image block in the original image, it is used to perform deduplication processing on the overlapping area between the image blocks; The respective image blocks are fused.

Optionally, different described quantization parameter sets correspond to different dynamic range intervals;

When determining the target quantization parameter set, the processor is configured to: determine the minimum interval to which the target dynamic range belongs; and determine the quantization parameter set corresponding to the minimum interval as the target quantization parameter set.

Optionally, the first interval is any interval in the intervals of the different dynamic ranges;

When determining the quantization parameter set corresponding to the first interval, the processor is configured to obtain a sample image whose dynamic range belongs to the first interval; and determine the quantization parameter set corresponding to the first interval according to the sample image.

Optionally, the quantization parameter set includes multiple quantization parameter subsets, and one of the quantization parameter subsets corresponds to a layer of the neural network;

When determining the quantization parameter set corresponding to the first interval according to the sample image, the processor is configured to obtain the input image of each layer of the neural network, wherein the input image of the input layer of the neural network is the input image of each layer of the neural network. The sample image, the input images of the remaining layers are the images obtained by the neural network using the sample image as the input for forward propagation; for each layer of the neural network, according to the dynamic range of the input image of the layer, Determine the subset of quantization parameters corresponding to this layer.

Optionally, the network parameters of the neural network are floating-point network parameters when the sample image is input.

Optionally, the processor determines the floating-point network parameters in the following manner, quantizes the original network parameters of the neural network to obtain fixed-point network parameters; performs inverse quantization on the fixed-point network parameters to obtain the floating-point network parameters. Click Network Parameters.

Optionally, when determining the minimum interval to which the target dynamic range belongs, the processor is configured to perform scaling processing on the target dynamic range; and determine the minimum interval to which the scaled target dynamic range belongs.

Optionally, the quantization parameter set includes multiple quantization parameter subsets, and one of the quantization parameter subsets corresponds to a layer of the neural network;

The processor is further configured to, for each layer of the neural network, perform quantization processing on the input image of the layer according to the quantization parameter subset corresponding to the layer, and input the quantized input image into the layer , wherein the input image of the input layer of the neural network is the image to be processed, and the input images of the other layers are the output images of the previous layer.

Optionally, each convolution kernel in the neural network has undergone quantization processing respectively.

Optionally, when performing quantization processing on the first convolution kernel, the processor is configured to perform quantization processing on the first convolution kernel according to a quantization parameter corresponding to the first convolution kernel, wherein the The first convolution kernel is any convolution kernel in the neural network.

Optionally, the network parameters of the neural network are fixed-point network parameters when the quantized image to be processed is input.

Optionally, the fixed-point network parameters are obtained by quantizing the original network parameters of the neural network.

Optionally, the fixed-point network parameters include weight parameters and/or bias parameters.

Optionally, the processor is further configured to preprocess the image input to the neural network.

Optionally, the preprocessing includes zero mean processing and/or normalization processing.

Optionally, the quantization parameter set includes a step size parameter and a zero point parameter.

Optionally, the image to be processed is an image block corresponding to the region of interest in the original image.

Optionally, when determining the region of interest, the processor is configured to perform semantic segmentation on the original image, determine object categories included in the original image; determine the degree of attention corresponding to each object category; The region corresponding to the object category with the highest degree is determined as the region of interest.

Optionally, the region of interest is selected by the user.

For the specific implementation of the image processing apparatuses in the various embodiments provided above, reference may be made to the relevant descriptions of the image processing methods provided above, which will not be repeated here.

In the image processing apparatus provided by the embodiment of the present application, different quantization parameters are set for input images with different dynamic ranges, so that, for the image to be processed, the quantization parameter suitable for the dynamic range of the image to be processed can be used for the image to be processed. The image data of the processed image is quantized, so that the fixed-point image data obtained after quantization can have sufficient accuracy on the basis of no distortion (that is, the value is consistent with the actual size), thereby reducing the accuracy loss caused by quantization. It meets the high-precision requirements of image regression tasks such as image denoising and super-resolution.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium; when the computer program is executed by a processor, any image processing method provided by the embodiments of the present application is implemented.

Embodiments of the present application may take the form of a computer program product implemented on one or more storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having program code embodied therein. Computer-usable storage media includes permanent and non-permanent, removable and non-removable media, and storage of information can be accomplished by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

As long as there is no conflict or contradiction in the various embodiments and implementations provided above, those skilled in the art can combine the various embodiments and implementations according to actual needs to form other embodiments or implementations. However, this application document is limited in space, and does not describe the embodiments or implementations formed by other combinations, but it can be understood that the solutions obtained by these combinations that do not require creative work also fall within the scope of the disclosure of the embodiments of this application. .

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also other not expressly listed elements, or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The methods, devices, and computer-readable storage media provided by the embodiments of the present application have been described in detail above. The principles and implementations of the embodiments of the present application are described in this document by using specific examples. The descriptions of the above embodiments are only for the purpose of Help to understand the methods and core ideas of the embodiments of the present application; at the same time, for those of ordinary skill in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation and application scope. , the content of this specification should not be construed as a limitation to the embodiments of the present application.

Claims (43)

An image processing method, characterized in that, applied to a neural network, the method comprising: determining the target dynamic range of the image to be processed, the image to be processed is used to input the neural network; determining a target quantization parameter set adapted to the target dynamic range from a plurality of candidate quantization parameter sets, wherein different said quantization parameter sets are adapted to different dynamic ranges; According to the target quantization parameter set, quantization processing is performed on the to-be-processed image, and the image data of the to-be-processed image is quantized into fixed-point data whose precision is adapted to the target dynamic range. The method according to claim 1, wherein the image to be processed is an image block in the original image; The method also includes: Perform fusion processing on the output images corresponding to each image block in the original image to obtain a target output image corresponding to the original image, wherein the output image is the image output by the neural network when the image block is used as input . The method according to claim 2, wherein the performing fusion processing on the output image corresponding to each image block in the original image comprises: De-duplication processing is performed on the overlapping area between the image blocks; Fusion is performed using the de-duplicated image blocks. The method according to claim 1, wherein the different quantization parameter sets correspond to different dynamic ranges, and the target quantization parameter set is determined based on the following methods: determining the minimum interval to which the target dynamic range belongs; The quantization parameter set corresponding to the minimum interval is determined as the target quantization parameter set. The method according to claim 4, wherein the first interval is any interval in the intervals of the different dynamic ranges, and the quantization parameter set corresponding to the first interval is determined based on the following methods: acquiring a sample image whose dynamic range belongs to the first interval; A quantization parameter set corresponding to the first interval is determined according to the sample image. The method according to claim 5, wherein the quantization parameter set includes a plurality of quantization parameter subsets, and one of the quantization parameter subsets corresponds to a layer of the neural network; The determining of the quantization parameter set corresponding to the first interval according to the sample image includes: Obtain the input image of each layer of the neural network, wherein the input image of the input layer of the neural network is the sample image, and the input images of the other layers are the neural network using the sample image as the input for forwarding disseminated images; For each layer of the neural network, a quantization parameter subset corresponding to the layer is determined according to the dynamic range of the input image of the layer. The method according to claim 6, wherein the network parameters of the neural network are floating-point network parameters when the sample image is input. The method according to claim 7, wherein the floating-point network parameters are determined based on the following methods: Quantifying the original network parameters of the neural network to obtain fixed-point network parameters; Perform inverse quantization on the fixed-point network parameters to obtain the floating-point network parameters. The method according to claim 4, wherein the determining the minimum interval to which the target dynamic range belongs comprises: scaling the target dynamic range; A minimum interval to which the scaled target dynamic range belongs is determined. The method according to claim 1, wherein the quantization parameter set includes a plurality of quantization parameter subsets, and one of the quantization parameter subsets corresponds to a layer of the neural network; The method also includes: For each layer of the neural network, perform quantization processing on the input image of the layer according to the quantization parameter subset corresponding to the layer, and input the quantized input image into the layer, wherein the neural network input The input image of the layer is the image to be processed, and the input images of the other layers are the output images of the previous layer. The method according to claim 10, wherein each convolution kernel in the neural network has undergone quantization processing respectively. The method according to claim 11, wherein the first convolution kernel is any convolution kernel in the neural network, and performing quantization processing on the first convolution kernel includes: The first convolution kernel is quantized according to the quantization parameter corresponding to the first convolution kernel. The method according to claim 1, wherein the network parameters of the neural network are fixed-point network parameters when the quantized image to be processed is input. The method according to claim 13, wherein the fixed-point network parameters are obtained by quantizing the original network parameters of the neural network. The method according to claim 13, wherein the fixed-point network parameters include weight parameters and/or bias parameters. The method according to claim 1, wherein the method further comprises: The image input to the neural network is preprocessed. The method according to claim 16, wherein the preprocessing comprises zero mean processing and/or normalization processing. The method according to claim 1, wherein the quantization parameter set includes a step size parameter and a zero point parameter. The method according to claim 1, wherein the image to be processed is an image block corresponding to a region of interest in the original image. The method of claim 19, wherein the region of interest is determined based on: Semantic segmentation is performed on the original image to determine the object category contained in the original image; Determine the degree of attention corresponding to each object category; The region corresponding to the object category with the highest degree of attention is determined as the region of interest. The method of claim 19, wherein the region of interest is selected by a user. An image processing device, characterized in that, applied to a neural network, comprising: a processor and a memory storing a computer program; The processor implements the following steps when executing the computer program: determining the target dynamic range of the image to be processed, the image to be processed is used to input the neural network; determining a target quantization parameter set adapted to the target dynamic range from a plurality of candidate quantization parameter sets, wherein different said quantization parameter sets are adapted to different dynamic ranges; According to the target quantization parameter set, quantization processing is performed on the to-be-processed image, and the image data of the to-be-processed image is quantized into fixed-point data whose precision is adapted to the target dynamic range. The apparatus according to claim 22, wherein the image to be processed is an image block in the original image; The processor is further configured to perform fusion processing on the output image corresponding to each image block in the original image to obtain a target output image corresponding to the original image, wherein the output image takes the image block as input the image output by the neural network. The device according to claim 23, wherein when the processor performs fusion processing on the output image corresponding to each image block in the original image, the processor is configured to remove the overlapping area between the image blocks. Reprocessing; using the de-repeated image blocks to perform fusion. The apparatus according to claim 22, wherein different quantization parameter sets correspond to different dynamic ranges; When determining the target quantization parameter set, the processor is configured to: determine the minimum interval to which the target dynamic range belongs; and determine the quantization parameter set corresponding to the minimum interval as the target quantization parameter set. The device according to claim 25, wherein the first interval is any interval in the intervals of different dynamic ranges; When determining the quantization parameter set corresponding to the first interval, the processor is configured to obtain a sample image whose dynamic range belongs to the first interval; and determine the quantization parameter set corresponding to the first interval according to the sample image. The apparatus according to claim 26, wherein the quantization parameter set comprises a plurality of quantization parameter subsets, and one of the quantization parameter subsets corresponds to a layer of the neural network; When determining the quantization parameter set corresponding to the first interval according to the sample image, the processor is configured to obtain the input image of each layer of the neural network, wherein the input image of the input layer of the neural network is the input image of each layer of the neural network. The sample image, the input images of the remaining layers are the images obtained by the neural network using the sample image as the input for forward propagation; for each layer of the neural network, according to the dynamic range of the input image of the layer, Determine the subset of quantization parameters corresponding to this layer. The apparatus according to claim 27, wherein the network parameters of the neural network are floating-point network parameters when the sample image is input. The apparatus according to claim 28, wherein the processor determines the floating-point network parameters in the following manner, quantizes the original network parameters of the neural network, and obtains fixed-point network parameters; The parameters are inversely quantized to obtain the floating-point network parameters. The device according to claim 25, wherein when determining the minimum interval to which the target dynamic range belongs, the processor is configured to perform scaling processing on the target dynamic range; and determining the scaled target The smallest interval to which the dynamic range belongs. The apparatus according to claim 25, wherein the quantization parameter set includes a plurality of quantization parameter subsets, and one of the quantization parameter subsets corresponds to a layer of the neural network; The processor is further configured to, for each layer of the neural network, perform quantization processing on the input image of the layer according to the quantization parameter subset corresponding to the layer, and input the quantized input image into the layer , wherein the input image of the input layer of the neural network is the image to be processed, and the input images of the other layers are the output images of the previous layer. The device according to claim 31, wherein each convolution kernel in the neural network has undergone quantization processing respectively. The apparatus according to claim 32, wherein when the processor performs quantization processing on the first convolution kernel, the first convolution kernel is quantized according to a quantization parameter corresponding to the first convolution kernel. The kernel performs quantization processing, wherein the first convolution kernel is any convolution kernel in the neural network. The apparatus according to claim 22, wherein the network parameters of the neural network are fixed-point network parameters when the quantized image to be processed is input. The apparatus according to claim 34, wherein the fixed-point network parameters are obtained by quantizing the original network parameters of the neural network. The apparatus according to claim 34, wherein the fixed-point network parameters include weight parameters and/or bias parameters. The apparatus according to claim 22, wherein the processor is further configured to preprocess the image input to the neural network. The apparatus according to claim 37, wherein the preprocessing includes zero mean processing and/or normalization processing. The apparatus according to claim 22, wherein the quantization parameter set includes a step size parameter and a zero point parameter. The apparatus according to claim 22, wherein the image to be processed is an image block corresponding to a region of interest in the original image. The device according to claim 40, wherein when determining the region of interest, the processor is configured to perform semantic segmentation on the original image, determine object categories included in the original image; The degree of attention corresponding to the object category; the area corresponding to the object category with the highest degree of attention is determined as the area of interest. The apparatus of claim 40, wherein the region of interest is selected by a user. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the image processing method according to any one of claims 1-21 is implemented.

Images