CN112116061A - Weight and activation value quantification method for long-term and short-term memory network - Google Patents

Abstract

The invention discloses a method for quantifying weights and activation values of long-term and short-term memory networks, comprising the steps of: 1) collecting a set of weight values and activation values of long-term and short-term memory networks; 2) determining a corresponding target quantization range, and starting Threshold traversal loop, according to the calculated scaling factor and saturation value under different thresholds, the weight value and activation value are scaled or set to saturation value; 3) After the traversal is completed, calculate the initial set of weight value and activation value respectively And the KL divergence of the number set after the mapping, and finally output the positive and negative direction truncation thresholds and the minimum KL divergence value. The invention realizes the transformation of the long-term and short-term memory network completed by high-precision floating-point number training into a fixed-point number network, and innovatively designs a quantization structure for the weight value and activation value of the long-term and short-term memory network. , reducing the hardware overhead and improving the running speed.

Description

A Quantization Method for Weights and Activations of Long Short-Term Memory Networks

technical field

The invention belongs to the field of cyclic neural network quantization, and in particular relates to a method for quantizing weights and activation values for long-term and short-term memory networks.

Background technique

With the continuous improvement of the computing power of graphics processors and general-purpose central processing units, the requirements of artificial neural networks for computational complexity are alleviated. Since 2012, artificial intelligence algorithms based on neural networks have continued to develop and are widely used in many fields such as pattern recognition, speech processing, and image processing. However, the development of hardware performance cannot always satisfy the evolution of algorithms. The SSD network proposed in 2016 introduced up to 50 billion computations, which needed to be run in large workstations. However, desktop processors and mobile processors cannot afford such a large amount of computation, which greatly limits the application scenarios of neural networks, including various terminal applications such as virtual reality technology and augmented reality technology. There are two solutions to this problem, one of which is to compress the redundant information in the neural network, such as quantizing the floating-point full-precision neural network, and compressing the data bit width to realize the compression of the network parameters. At present, most quantization schemes are aimed at the convolutional neural network in the neural network, because of its many network parameters and high computational complexity, there is no quantization scheme for the recurrent neural network. Memory network (LSTM), designed a set of quantization schemes suitable for small-scale LSTM networks, which can reduce the overhead of hardware design while ensuring network accuracy, so that LSTM networks can be applied in mobile devices at low cost, and promote neural-based The development of artificial intelligence algorithms for networks.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for quantifying the weights and activation values of long-term and short-term memory networks, which converts the long-term and short-term memory networks trained with high-precision floating-point numbers into fixed-point number networks, including the quantization of weight values and activation values. The fixed-point number network has similar network precision to the floating-point number network, and can be applied to the terminal system at low cost and high speed.

The present invention adopts following technical scheme to realize:

A quantification method for the weights and activations of long short-term memory networks, including the following steps:

1) Collect the weight value set of the long and short-term memory network, and determine the target quantization range of the weight value, start the threshold traversal cycle, and scale or set the weight value according to the calculated scaling factor and saturation value under different thresholds. Saturation value; after the traversal is completed, calculate the KL divergence of the initial set of weight values and the mapped set, and finally output the positive and negative direction truncation thresholds and the minimum KL divergence value respectively;

2) Collect the activation value set of the long short-term memory network, and determine the initial quantization range and target quantization range of the activation value, start the threshold traversal cycle, and calculate the activation value according to the calculated scaling factor and saturation value under different thresholds. Scale or set to saturation value; after the traversal is completed, calculate the KL divergence of the initial set of activation values and the mapped set, and finally output the positive and negative direction truncation thresholds and the minimum KL divergence value respectively.

A further improvement of the present invention is that the concrete realization method of step 1) is as follows:

101) The weight value of the long short-term memory network The loop parameter or input parameter in the LSTM network, the weight value set size is small and sparse, ranging from 100 to 4500;

102) For the input parameters of the LSTM network of the blocking sequence, first search for the appropriate target quantization range num for the weight value, and select a total of 2 ^N values for num. For the KL divergence of the quantization scheme, the quantization selection of the input parameter here is INT3, that is, num is equal to 8; and the loop parameter is larger than the input parameter data, so the quantization selection of the loop parameter here is INT4, that is, num is equal to 16;

103) Then traverse and search for the best threshold in INT3, and the minimum grouping step step of the weight value is set to "(max-min)/num/3"; the process of "traversing all thresholds" is equivalent to 3 × num initial In the grouping, the data of num groups is selected to be scaled according to the scaling factor sf, and the data outside num groups is set to the saturation value low and high; it is traversed from the 3 × num initial groups and controlled by the loop variables j and g; when the negative number The direction threshold takes the second group, that is, the average value of idk=2, and the positive direction threshold takes the second group, that is, the average value of idj=2, and the KL divergence is the smallest. When searching for the best threshold in INT4 traversal , when the negative direction threshold takes the sixth group, that is, the center value of idk=6, and the positive direction threshold takes the fifth group, that is, the center value of idj=5, the KL divergence is the smallest;

104) INT3 quantization finally divides the input parameters into 24 groups, that is, 3 × num groups. According to the obtained optimal threshold, the input parameters in the first, second, and 23rd and 24th groups are eliminated and set as Negative direction threshold and positive direction threshold; and INT4 quantization finally divides the loop parameters into 48, that is, 4 × num groups according to the obtained optimal threshold, the weights within the range of the 1st to 6th groups and the 44th to 48th groups are divided Eliminate and set as negative direction threshold and positive direction threshold respectively.

The further improvement of the present invention is, the concrete realization method of step 2) is as follows:

201) The activation value of the long short-term memory network is iteratively run 1000 times by the LSTM network, such as the output data of the multiplication array of the gate module, the size of the activation value set is not limited, and it is relatively dense, which is about 106;

202) For the activation value of the LSTM network, first search for the initial non-precision loss quantization range num1 of the activation value and the appropriate target quantization range num, and after calculating with the same specific quantization algorithm, num, that is, INT8 is equal to 128, num1, that is, INT16 is equal to 65536, Collect the KL divergence of the optimal quantization scheme under each group of num1 and num;

203) Then traverse the INT8 to search for the best threshold, the minimum grouping step step of the activation value is set to "(max-min)/num1", the traversal is only controlled by the loop variable k, and the maximum traversal range of the threshold is the initial grouping num1 The bit width difference value of the target group num, for INT16 and INT8, the value range is an integer between 0 and 8; since the threshold traversal of the activation value is only controlled by one variable, the quantization scheme of idk=1 is selected here;

204) Finally, the activation parameters are divided into 1024 groups, that is, 8×num groups. According to the obtained optimal threshold, the activation values in the range of the 1st to 256th groups and the 512th to 1024th groups are eliminated and set to the negative direction respectively. Threshold and positive direction threshold.

The present invention at least has the following beneficial technical effects:

The neural network learns the pattern separability of the data samples during the training process, and at the same time, due to the noise existing in the data, the network has strong robustness. That is, making a small change in the input sample does not affect performance drastically. The introduction of noise will change the activation value output of each layer, but it has little effect on the result, that is to say, the trained network has a certain tolerance for these noises. Using high precision in the training process, such as the numerical representation of FP16, makes the network have a certain tolerance. If low-precision data is used to represent network parameters and activation values during inference, there will be errors, and this error is also a kind of noise. Therefore, low-precision data, such as fixed-point INT8, introduces differences that are within the tolerance of the network and will not have much impact on the results. Finally, in the LSTM network predictor, the network activation value is represented by INT8 data, and the weight is represented by INT4 and INT3. Compared with the Golden Reference, the accuracy of the LSTM predictor has a very small drop in accuracy, and the change is controlled from -0.06% to +0.01% between. The results show that the quantization method designed in this paper is suitable for the cache replacement algorithm based on LSTM network, which reduces the hardware overhead and improves the running speed while ensuring the hardware implementation accuracy of the algorithm. The goal of the present invention is to quantify the trained FP16 LSTM network into INT8 or even INT4. The quantization operation is for each component in the LSTM network, such as the multiplier array of the gate module, the Sigmoid activation function of the network module, etc., to obtain the most suitable output bit width of each module. The specific scheme is to optimize the TensorRT INT8 convolutional neural network quantization scheme according to the activation value and weight distribution of the LSTM network, and propose a set of quantization methods suitable for the LSTM network. In general, the present invention has the following characteristics:

1. The quantization scheme distinguishes between the weight value and the activation value in the LSTM network, the weight value is quantized offline, and the activation value is quantized online.

2. The quantization scheme is based on the linear quantization scheme and is optimized for the distribution of LSTM network data.

The present invention has the following advantages:

1. The weight value is quantized offline in the present invention, which reduces the data bit width of the weight value as much as possible; the activation value is quantized online to prevent the quantization scheme from bringing additional hardware resource overhead.

2. The present invention adopts a linear, non-saturated quantization method, takes the KL divergence as a standard, searches for a threshold value in a global range, sets the numerical value whose data distribution exceeds the threshold value as a saturated value, and converts the scattered distribution into a dense distribution, The quantization error is reduced.

3. The present invention corrects the problem that sparse data leads to errors in KL divergence calculation in the LSTM network, so that the solution can be adapted to LSTM networks of different sizes.

Description of drawings

Figure 1 is a hardware architecture diagram of a long short-term memory network;

Fig. 2 is a schematic diagram of a saturated quantization process designed by the present invention;

Fig. 3 is the quantization process pseudo code of weight value and activation value designed by the present invention;

Fig. 4 is the hardware structure diagram of the weight value quantization designed by the present invention;

Fig. 5 is the hardware structure diagram of activation value quantization designed by the present invention;

Fig. 6 is the change trend diagram of KL divergence with the integer range when INT3 is quantized;

Figure 7 is a graph showing the variation trend of KL divergence with idk and idj during INT3 quantization;

Fig. 8 is a schematic diagram of reserved grouping and excluded grouping during INT3 quantization;

Figure 9 is a schematic diagram of the comparison between INT3 quantization and FP16.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

In the quantification method oriented to the long and short-term memory network proposed by the present invention, the hardware of the long-term and short-term memory network is shown in Figure 1, which consists of three parts: a gate module, a network module and a storage module. The gate module computes the input gate vector, output gate vector, forget gate vector and memory cell vector in the LSTM network. The network module is responsible for the activation function calculation, network state calculation and output vector calculation in the LSTM network. The storage module is responsible for storing the parameters and states of the LSTM network.

The quantization scheme proposed by the present invention adopts a saturated quantization method. Set a threshold value T to map the data in the interval -|T| to |T| to -128 to 127 in FP16. The value range of the threshold T is 0<T<65504. Data outside the threshold range is mapped to a saturation value of -128 or 127. The schematic diagram is shown in Figure 2. When the threshold is properly selected, scattered and large activation values or weight values can be eliminated to ensure the mapping accuracy.

The calculation of the threshold value T in the saturated quantization scheme proposed by the present invention is as follows: first, the data distribution of FP16 is set as the standard, and when INT8 is used to represent this optimal distribution, it is equivalent to re-encoding of information, and then the relative entropy ( Elativeentropy), also known as KL divergence, to measure the difference between two distributions, the greater the relative entropy, the greater the difference between the two distributions, and vice versa. Then, all the thresholds are traversed, and the KL divergence is calculated. Finally, the threshold with the smallest KL divergence is selected, which represents that under this threshold, the saturated linear quantization method is used for mapping, and the distribution difference between FP16 and INT8 is the smallest.

The calculation formula of KL divergence in the saturated quantization scheme proposed by the present invention is as (1). q represents the data distribution under the optimal encoding method, such as FP16. p represents the recoded data distribution, such as INT8. KL divergence represents the difference between two distributions, and its value must be greater than zero. However, when the data in the dataset is relatively sparse, such as the weight parameter, p(x) or q(x) may be 0. Due to the existence of the log function, the calculation of the KL divergence will be wrong. In order to prevent this from happening , when p(x) or q(x) is 0, assign it as EPS, that is, MATLAB single floating point precision, and in order to ensure that the calculated KL divergence is greater than zero, according to Gibbs inequality, ∑p or ∑ q must be 1, so pick a random distribution and subtract an EPS value.

if p(x)or q(x)=0, p(x)or q(x)=+EPS

p or q(rand)=p or q(rand)-EPS (1)

The pseudo code of the quantization scheme proposed by the present invention is as shown in Figure 3, and the implementation steps are as follows:

1. Collect activation or weight values. The weight value is the loop parameter or input parameter in the LSTM network; the activation value is the output data of each module (such as the multiplication array of the gate module) after 1000 iterations of the LSTM network. The LSTM network weight values or activation values of different sequences are quantized separately; the weight value set is small and sparse, around 100-4500. The size of the activation value data set is not limited, and it is relatively dense. The size of the data set in this paper is about 106.

2. Determine the threshold traversal range set sum, and start the threshold traversal loop. Calculate the scaling factor sf and saturation values high and low under different thresholds;

3. Traverse the activation value or weight set sum, scale each activation value or weight value according to the scaling factor and saturation value calculated in step 2, or set it to a saturation value;

4. Complete the activation value or weight value traversal, calculate the KL divergence of the initial data set x and the mapped data set x_tmp, and compare it with the result of the previous iteration, and retain the result with a smaller KL divergence. The idk output by the final algorithm represents the truncation threshold in the negative direction, idj represents the truncation threshold in the positive direction, and the minimum KL divergence value.

Performance Testing of the Invention

According to the weight quantization method of the example, the parameters in the LSTM network are sequentially quantized, the input parameters are represented by the numerical value of INT3, and the loop parameter is represented by the numerical value of INT4. According to the activation value quantization method of the example, each module in the LSTM network is quantized in turn. This process is called calibration. The results are shown in Table 2. For the Sigmoid activation function module, when all the precision is preserved, the output bit width is 25 bits, and after quantization, the output bit width is truncated to 10 bits. It shows that the activation value of the sigmoid function is largely distributed around the threshold, which is determined by the characteristics of the sigmoid function. Therefore, after a large number of values are set to saturation values, the error is still relatively small. For the Sigmoid activation function module, the quantization hardware structure of the activation value is shown in Figure 5. After the high-order [25:12] line AND, the control selector outputs the low-order [11:2] or the saturation value "512", if the MSB is " 1", the complement code is output. In the same way, for the input parameters of the multiplier B module, the quantization hardware structure of its weight is shown in Figure 4. After the high-order [18:8] line AND, the control selector outputs the low-order [7:0] or saturation value" 128", if the MSB is "1", the complement is output.

After calibration, the accuracy rate of each LSTM hardware is compared with that of the software algorithm FP16 as shown in Table 1. The accuracy rate drops very little, no more than 0.06%, and even the accuracy rate of some predictors is improved by 0.01%. The results show that the quantization method designed in this paper is suitable for a specific LSTM network algorithm, which reduces the hardware overhead and improves the running speed while ensuring the hardware implementation accuracy of the algorithm.

Table 1. Comparison of hardware accuracy and software algorithm FP16 accuracy after quantization of various LSTM activation values

Table 2. Quantization results of each module of the LSTM network in the present invention

Example

The present invention can be implemented in a software and hardware system of a long short-term memory network.

Taking the input parameters of the LSTM network as an example, the weight value quantization is performed in three steps:

The first step is to search for a suitable target quantization range num. During the experiment, a total of 2 ^N values were selected, where N = 1 to 7, and the kl_comp_weight() function was run to collect the KL divergence under the optimal quantization scheme for each num. The results are shown in Figure 6. When num increases, the KL divergence decreases, that is, when the bit width of the fixed-point number is larger, the distribution of the fixed-point number and the floating-point number is closer, and the error is smaller. When num is greater than 8, the KL divergence decreases rapidly with the increase of num. In order to reduce the cost of hardware design, the quantization of input parameters is selected as INT3.

The second step is to search for the best threshold in the case of INT3, that is, when num is equal to 8. The search process of the threshold is shown in Figure 7. The figure shows the KL divergence of fixed-point and floating-point numbers after quantization under different thresholds. When the negative direction threshold takes the second group, that is, the average value of idk=2, and the positive direction threshold takes the second group, that is, the average value of idj=2, the KL divergence is the smallest, with a value of 0.064.

In the third step, as shown in Figure 8, the input parameter distribution is divided into 24, that is, 3 × num groups are drawn into a histogram. According to the optimal threshold obtained in Figure 7, the input parameters in the range of the 0th, 1st group and the 23rd, 24th group are eliminated, and set as the negative direction threshold and the positive direction threshold respectively.

Finally, the distribution of the data of FP16 before quantization and INT3 after quantization arranged according to size is shown in FIG. 9 . It can be seen that the range of the input parameters before quantization is about -0.39 to 0.45. After the quantization method kl_comp_weight() is executed, due to the scattered distribution of the two data points in the 0th group and the 24th group, as shown in Figure 9. The points in the solid circle , and exceeds the threshold, so it is set to the saturation value. The dense data in the distribution center is more accurately quantified as INT3.

Claims (3)

1. a quantization method for the weight and activation value of long short-term memory network, is characterized in that, comprises the following steps: 1) Collect the weight value set of the long and short-term memory network, and determine the target quantization range of the weight value, start the threshold traversal cycle, and scale or set the weight value according to the calculated scaling factor and saturation value under different thresholds. Saturation value; after the traversal is completed, calculate the KL divergence of the initial set of weight values and the mapped set, and finally output the positive and negative direction truncation thresholds and the minimum KL divergence value respectively; 2) Collect the activation value set of the long short-term memory network, and determine the initial quantization range and target quantization range of the activation value, start the threshold traversal cycle, and calculate the activation value according to the calculated scaling factor and saturation value under different thresholds. Scale or set to saturation value; after the traversal is completed, calculate the KL divergence of the initial set of activation values and the mapped set, and finally output the positive and negative direction truncation thresholds and the minimum KL divergence value respectively. 2. a kind of quantization method for the weight of long short-term memory network and activation value according to claim 1, is characterized in that, the concrete realization method of step 1) is as follows: 101) The weight value of the long short-term memory network The loop parameter or input parameter in the LSTM network, the weight value set size is small and sparse, ranging from 100 to 4500; 102) For the input parameters of the LSTM network of the blocking sequence, first search for the appropriate target quantization range num for the weight value, and select a total of 2 ^N values for num. For the KL divergence of the quantization scheme, the quantization selection of the input parameter here is INT3, that is, num is equal to 8; and the loop parameter is larger than the input parameter data, so the quantization selection of the loop parameter here is INT4, that is, num is equal to 16; 103) Then traverse and search for the best threshold in INT3, and the minimum grouping step step of the weight value is set to "(max-min)/num/3"; the process of "traversing all thresholds" is equivalent to 3 × num initial In the grouping, the data of num groups is selected to be scaled according to the scaling factor sf, and the data outside num groups is set to the saturation value low and high; it is traversed from the 3 × num initial groups and controlled by the loop variables j and g; when the negative number The direction threshold takes the second group, that is, the average value of idk=2, and the positive direction threshold takes the second group, that is, the average value of idj=2, and the KL divergence is the smallest. When searching for the best threshold in INT4 traversal , when the negative direction threshold takes the sixth group, that is, the center value of idk=6, and the positive direction threshold takes the fifth group, that is, the center value of idj=5, the KL divergence is the smallest; 104) INT3 quantization finally divides the input parameters into 24 groups, that is, 3 × num groups. According to the obtained optimal threshold, the input parameters in the first, second, and 23rd and 24th groups are eliminated and set as Negative direction threshold and positive direction threshold; and INT4 quantization finally divides the loop parameters into 48, that is, 4 × num groups according to the obtained optimal threshold, the weights within the range of the 1st to 6th groups and the 44th to 48th groups are divided Eliminate and set as negative direction threshold and positive direction threshold respectively. 3. a kind of quantization method for the weight of long short-term memory network and activation value according to claim 1, is characterized in that, the concrete realization method of step 2) is as follows: 201) The activation value of the long short-term memory network is iteratively run 1000 times by the LSTM network, such as the output data of the multiplication array of the gate module, the size of the activation value set is not limited, and it is relatively dense, which is about 106; 202) For the activation value of the LSTM network, first search for the initial non-precision loss quantization range num1 of the activation value and the appropriate target quantization range num, and after calculating with the same specific quantization algorithm, num, that is, INT8 is equal to 128, num1, that is, INT16 is equal to 65536, Collect the KL divergence of the optimal quantization scheme under each group of num1 and num; 203) Then traverse the INT8 to search for the best threshold, the minimum grouping step step of the activation value is set to "(max-min)/num1", the traversal is only controlled by the loop variable k, and the maximum traversal range of the threshold is the initial grouping num1 The bit width difference value of the target group num, for INT16 and INT8, the value range is an integer between 0 and 8; since the threshold traversal of the activation value is only controlled by one variable, the quantization scheme of idk=1 is selected here; 204) Finally, the activation parameters are divided into 1024 groups, that is, 8×num groups. According to the obtained optimal threshold, the activation values in the range of the 1st to 256th groups and the 512th to 1024th groups are eliminated and set to the negative direction respectively. Threshold and positive direction threshold.

Images