CN116611470A - Method for setting quantization bit number of neural network model - Google Patents

Abstract

The present invention relates to the technical field of model quantization, specifically a method for setting quantization digits of a neural network model, comprising the following steps: preparing a data set containing a small amount of data; defining quantization loss evaluation indicators; defining model quantization digits; obtaining model-related memory limits ; define the target neural network model; calculate the quantization loss of different quantization digits of the activation value; analyze the model reasoning step and the activation value in the memory at each moment; the beneficial effect is: the neural network model quantization digit setting method proposed by the present invention solves the problem of memory limitation Conditional activation value quantization digit selection problem. Firstly, analyze the precision loss of activation values under different bit widths, and then analyze the set of activation values in memory at different times to construct an optimization problem. Under the condition of limiting the memory usage of activation values at all times, through integer linear programming method, optimize to get Optimal precision loss.

Description

A method for setting quantization digits of neural network model

technical field

The invention relates to the technical field of model quantization, in particular to a method for setting quantization digits of a neural network model.

Background technique

With the continuous development of deep learning related technologies, neural network models have been widely used in many industries and scenarios. However, the deep learning model has a large amount of parameters and calculations, and has high requirements for hardware resources, which often conflict with the limitations of hardware resources.

In the existing technology, there are a large number of embedded devices in the Internet of Things field. Due to the extremely limited hardware resources of the devices, it is difficult to deploy deep learning models. In addition, the deep learning model is getting bigger and bigger, even on the server or cloud, the pressure to deploy some large models is gradually highlighted. Among them, memory limitation is a key factor restricting model deployment, which determines whether the model can be deployed to the device.

However, during the inference process of deep learning models, it is generally the activation values that occupy most of the memory. Model quantization is an effective method to reduce the memory usage of models, especially multi-precision model quantization. However, compared with traditional quantization methods, multi-precision quantization often leads to a greater loss of model accuracy. How to choose different quantization digits for different weights and activation values is the key to affecting the loss of model accuracy. At present, there are some methods for multi-precision quantization of model weights, which do not involve the selection of activation value digits. However, some are selected for the number of activation values, and do not consider the goal of reducing the accuracy of the model.

Contents of the invention

The purpose of the present invention is to provide a method for setting the number of quantization digits of a neural network model to solve the problems raised in the above-mentioned background technology.

In order to achieve the above object, the present invention provides the following technical solutions: a method for setting the number of quantized digits of a neural network model, said method comprising the following steps:

Prepare datasets containing small amounts of data;

Define quantitative loss assessment indicators;

Define the number of quantized digits of the model;

Get the memory limit related to the model;

Define the target neural network model;

Calculate the quantization loss of different quantization digits of the activation value;

Analyze model reasoning steps and activation values in memory at each moment;

The optimization problem is constructed and solved using the integer linear programming method;

Model quantization and inference.

Preferably, when preparing a data set containing a small amount of data, the data set is used as a calibration data set and a quantization loss calculation data set.

Preferably, when defining the quantitative loss evaluation index, the difference between the non-quantized activation value data and the quantized activation value data is calculated, and the difference is calculated by but not limited to mean square error, cosine similarity, etc., as the quantized loss.

Preferably, when defining model quantization digits, based on hardware support and model accuracy requirements, model quantization digits are but not limited to 2, 4, 6, and 8 digits, and Bitwidth is used to represent the set of quantization digits.

Bitwidth = {2, 4, 6, 8}.

Preferably, when obtaining the model-related memory limit, during the model inference process, the inference-related memory usage includes the inference framework memory usage, code data memory usage, model weight memory usage, and activation value memory usage, where the model-related memory usage mainly includes model weight memory Occupancy and activation value memory occupation; suppose the hardware memory size is m _total , the memory other than the model-related memory occupation is m _other , and the remaining memory size is m _model , which is the model-related memory limit.

Preferably, when defining the target neural network model, the neural network model is a mainstream neural network model including but not limited to ResNet, MobileNet, SSD.

Preferably, when calculating the quantization loss of different quantization digits of the activation value, for each sample data in the loss calculation data set, use the pre-quantization model for reasoning, and obtain all the activation value data, and for all the activation value data, use all Quantization digits, calculate the quantization loss of different quantization digits of each activation value, after completing the inference of all sample data, summarize the quantization losses of different quantization digits of each activation value, and use but not limited to the mean value to calculate the quantization loss of all sample data The quantization loss set Loss of different quantization bits of each activation value; the formula is as follows, where loss _ab represents the quantization loss of activation value data a using quantization bits b, Loss={loss _ab }, a∈Activations,b∈Bitwidth.

Preferably, when analyzing the model reasoning steps and the activation values in the memory at each moment, during the model reasoning process, the model weights are always stored in the memory, and the activation values that are no longer used will be destroyed in time to release the occupied memory space; let Activations represent All the activation values of the model, activation_set _t represents the set of activation values in memory at time t, so any activation _sett is a subset of Activations, and at different times The elements in will be repeated; through step-by-step reasoning, the activation value set Activation _sets in the memory at all times can be obtained, the formula is as follows, where T represents the set of all moments in the inference process.

Preferably, when the optimization problem is constructed and solved using the integer linear programming method, the optimization objective is to minimize the total quantization loss, and the constraint condition is that the memory occupied by the activation value set at each moment is less than the activation value memory limit. Since the model weight occupies memory It can be calculated in advance, assuming m _weight , let m _c represent the activation value memory limit, use the following formula to calculate m _c ,

m _c = m _model - m _weight

Let total_loss represent the total quantization loss, m _t represents the memory size occupied by the activation value set at time t, m _a represents the memory size occupied by the activation value a, size _a represents the number of elements of the activation value a, b _a represents the activation value a quantization bits, Obtained from the Loss collection,

min total_loss

m _a = size _a *b _a

The optimization problem is an integer linear programming problem, and the optimization variable is the quantization bit b _a of each activation value, which is solved by using the integer linear programming method to obtain the quantization bit number of each activation value.

Preferably, during model quantization and inference, the optimized quantization bits are used to perform model quantization and then perform model inference.

Compared with prior art, the beneficial effect of the present invention is:

The method for setting the quantization digits of the neural network model proposed by the invention solves the problem of selecting the quantization digits of the activation value under the condition of memory limitation. Firstly, analyze the precision loss of activation values under different bit widths, and then analyze the set of activation values in memory at different times to construct an optimization problem. Under the condition of limiting the memory usage of activation values at all times, through integer linear programming method, optimize to get Optimal precision loss. Through the above method, the optimal activation value quantization bit can be selected under the condition of hardware memory limitation, so that the precision loss is minimized, and it has high practical value and innovative value.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention.

Detailed ways

In order to clearly and completely describe the purpose, technical solution, and advantages of the present invention, the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are part of the embodiments of the present invention, rather than all embodiments, and are only used to explain the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. All other embodiments obtained under the premise of creative labor belong to the protection scope of the present invention.

Please refer to Fig. 1, the present invention provides a kind of technical scheme: a kind of quantization digit setting method of neural network model, described method comprises the following steps:

First, prepare a dataset containing a small amount of data. This dataset serves as a calibration dataset as well as a quantization loss computation dataset.

Second, define quantitative loss evaluation metrics. Quantized loss calculation is to calculate the difference between non-quantized activation value data and quantized activation value data. The difference can be calculated by using but not limited to mean square error, cosine similarity, etc., as quantized loss. Non-quantized activation value data refers to the use of floating-point weights and inputs (previous activation value data) to calculate corresponding operators, such as convolution, full connection, etc., and the calculation result is non-quantized activation value data. Quantizing the activation value data refers to quantizing the weight of the floating-point number and the input data respectively, using the quantized data to calculate the corresponding operator, and dequantizing the calculated result to obtain the quantized activation value data.

Third, define the model quantization digits. Based on hardware support and model accuracy requirements, the model quantization digits can be but not limited to 2, 4, 6, and 8 digits, and Bitwidth is used to represent the set of quantization digits.

Bitwidth={2, 4, 6, 8}

Fourth, get the model-related memory limit. During model inference, inference-related memory usage includes inference framework memory usage, code data memory usage, model weight memory usage, and activation value memory usage, among which model-related memory usage mainly includes model weight memory usage and activation value memory usage. Assume that the hardware memory size is m _total , the memory other than the model-related memory occupation is m _other , and the remaining memory size is m _model , that is, the model-related memory limit.

m _model = m _total - m _other

Fifth, define the target neural network model. The neural network model includes but is not limited to ResNet, MobileNet, SSD and other mainstream neural network models.

Sixth, calculate the quantization loss of different quantization digits of the activation value. For each sample data in the loss calculation dataset, use the pre-quantization model for inference, and obtain all activation value data. At the same time, for all activation value data, all quantization bits are used to calculate the quantization loss of different quantization bits for each activation value. After the inference of all sample data is completed, the quantization losses of different quantization digits of each activation value are summarized, and the quantization loss set Loss of different quantization digits of each activation value of all sample data is calculated by means of but not limited to the mean value. The formula is as follows, where loss _ab represents the quantization loss of activation value data a using quantization bits b.

Loss＝{loss _ab },a∈Activations,b∈Bitwidth

Seventh, analyze the model reasoning steps and the activation values in memory at each moment. During the model inference process, the model weights are always stored in memory, and the activation values that are no longer used will be destroyed in time to release the occupied memory space. Let Activations represent all the activation values of the model, and activation_set _t represents the set of activation values in memory at time t, so any They are all subsets of Activations, but at different times /> Elements in may be repeated. Step by step reasoning, the activation value set Activation _sets in the memory at all moments is obtained, the formula is as follows, where T represents the set of all moments in the reasoning process.

Eighth, the optimization problem is constructed and solved using the integer linear programming method. The optimization goal is to minimize the total quantization loss, and the constraint condition is that the memory occupied by the activation value set at each moment is less than the activation value memory limit. Since the memory occupied by the model weight can be calculated in advance, assuming it is m _weight , let m _c represent the memory limit of the activation value, and use the following formula to calculate m _c .

m _c = m _model - m _weight

Let total_loss represent the total quantization loss, m _t represents the memory size occupied by the activation value set at time t, m _a represents the memory size occupied by the activation value a, size _a represents the number of elements of the activation value a, b _a represents the activation value a quantization bits. Available from the Loss collection.

min total_loss

m _a = size _a *b _a

The above optimization problem is an integer linear programming problem, and the optimization variable is the quantization digit b _a of each activation value, which can be solved by an integer linear programming method to obtain the quantization digit of each activation value.

Ninth, model quantification and inference. Use the optimized quantization digits to perform model quantization and then perform model inference.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (10)

1. A method for setting quantization bit number of neural network model is characterized in that: the method comprises the following steps:

preparing a dataset comprising a small amount of data;

defining a quantization loss evaluation index;

defining a model quantization bit number;

acquiring a related memory limit of a model;

defining a target neural network model;

calculating quantization losses of different quantization bits of the activation value;

analyzing the model reasoning step and the activation values in the memory at each moment;

constructing an optimization problem, and solving by adopting an integer linear programming method;

model quantization and reasoning.

2. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when preparing a data set containing a small amount of data, the data set is used as a calibration data set and a quantization loss calculation data set.

3. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when the quantization loss evaluation index is defined, the non-quantization activation value data and the difference of the quantization activation value data are calculated, and the difference is calculated by adopting, but not limited to, mean square error, cosine similarity and the like as the quantization loss.

4. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when defining the model quantization bit number, based on the hardware support condition and the model precision requirement, the model quantization bit number is but not limited to 2,4,6,8 bit number, using Bitwidth to represent the quantization bit number set,

Bitwidth＝{2，4，6，8}。

5. the method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when the model related memory limit is acquired, in the model reasoning process, reasoning related memory occupation comprises reasoning frame memory occupation, code data memory occupation, model weight memory occupation and activation value memory occupation, wherein the model related memory occupation mainly comprises model weight memory occupation and activation value memory occupation; assume that the hardware memory size is m _total The memory outside the related memory occupation of the model is m _other Remaining memory size m _model I.e. model dependent memory limitations.

6. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when defining the target neural network model, the neural network model is a mainstream neural network model including, but not limited to ResNet, mobileNet, SSD.

7. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when calculating quantization losses for different number of quantization bits of the activation value, each sample in the data set is calculated for the lossThe method comprises the steps of (1) reasoning data by using a pre-quantization model, obtaining all activation value data, calculating quantization losses of different quantization digits of each activation value by adopting all quantization digits for all activation value data, summarizing the quantization losses of different quantization digits of each activation value after all sample data reasoning is completed, and calculating a quantization Loss set Loss of different quantization digits of each activation value of all sample data by adopting a mean value mode and the like; the formula is as follows, wherein loss _ab Represents quantization Loss of the activation value data a using the quantization bit number b, loss= { Loss _ab },a∈Activations,b∈

Bitwidth。

8. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: when analyzing the model reasoning step and the activation values in the memory at all times, in the model reasoning process, the model weight is always stored in the memory, and the activation values which are not used any more can be destroyed in time, so that the occupied memory space is released; let the actions represent all the activation values of the model, activation_set _t Representing the set of activation values in memory at time t, and therefore eitherAre all subsets of actions, but +.>The elements of (a) are repeated; gradually reasoning to obtain the Activation value set Activation in the memory at all moments _sets The formula is as follows> Where T represents the set of all moments in the reasoning process.

9. According to claim 1The method for setting the quantization bit number of the neural network model is characterized by comprising the following steps of: when the optimization problem is constructed and solved by adopting an integer linear programming method, the optimization target is made to be minimum in total loss, the constraint condition is that the memory occupied by the activation value set at each moment is smaller than the memory limit of the activation value, and the memory occupied by the model weight can be calculated in advance and is assumed to be m _weight Let m _c Representing an activation value memory constraint, m is calculated using the following formula _c ，

m _c ＝m _model -m _weight

Let total_loss denote total loss, m _t Representing the memory size, m occupied by the time t activation value set _a Representing the memory size and size occupied by the activation value a _a The number of elements representing the activation value a, b _a Quantization bit number, loss, representing activation value a _aba From the set of Loss,

min total_loss

m _a ＝size _a *b _a

the optimization problem is an integer linear programming problem, and the optimization variable is the quantization bit number b of each activation value _a And solving by adopting an integer linear programming method to obtain the quantization bit number of each activation value.

10. The method for setting quantization bit numbers of a neural network model according to claim 1, wherein: during model quantization and reasoning, model quantization is performed by using the quantization bit number obtained by optimization, and model reasoning is further performed.