WO2025089499A1 - Data-type recognition weight scaling apparatus and method for floating point quantization - Google Patents

Abstract

Provided are a data-type recognition weight scaling apparatus and a method for floating-point quantization. According to an embodiment of the present invention, the weight scaling apparatus comprises: a scaler calculation unit for calculating a weight scaler value for adjusting the scale of a weight by recognizing a data type; and a multiplication unit for multiplying the calculated weight scaler by the weight. The weight scaler is determined by the ratio of the value of the standard deviation of a weight of a corresponding layer and the value of the standard deviation of a weight of the layer following the corresponding layer.

Description

Data-type-aware weight scaling device and method for floating-point quantization

The present invention relates to a data-type aware weight scaling device and method for floating point quantization.

Deep learning models often use depthwise-separable convolution blocks (DS Conv Blocks) to reduce the number of learnable parameters (weights) and operations (FLOPs).

The depth-wise separable convolution block is composed of Pointwise Convolution (PW Conv) - Depthwise Convolution (DW Conv) - Pointwise Convolution (PW Conv) in that order, and each convolution layer (conv) is usually followed by a BatchNorm layer (BN) and a non-linear function such as ReLU.

Deep learning models use floating-point-based FP32 and FP16 data types for precise calculations. Recently, GPUs and NPUs apply 16-bit or 8-bit quantization, which reduces memory footprint and has the advantage of accelerating calculations for efficient inference.

The 8-bit quantization method (INT8) based on integers, which was widely used in the past, is being widely adopted, and the floating-point-based quantization method, which has the same memory footprint but shows superior performance.

Floating point types are divided into sign, exponent, and mantissa parts. Currently, there are various formats for floating point format data types less than 8 bits or less than 16 bits, as there is no international standard from IEEE. For example, the FP8 (1-4-3) format consists of 1 bit _sign , 4 bit _exponent , and 3 bit _mantissa .

Typically, quantization is applied to both weights and activation, and the formula is as follows:

로 설정한다.The bias _default (hereinafter referred to as "default bias") can be set differently for each user. In addition, the FP8 format has a very limited expression range. To solve this, the expression range of the FP8 format can be adjusted by adding a bias _extra (hereinafter referred to as "additional bias") term as in Equation 2. In this patent, the default bias is

Set to .

The expressible range can be adjusted by adjusting the additional bias (bias _extra ), which can reduce the quantization error. For example, in FP8(1-4-3), when the basic bias is 7, the fixed quantization range of the positive region is 0.0087890625 to 480 when bias _extra = 0, and when bias _extra = 4, the fixed quantization range of FP8(1-4-3) is 0.00054931640625 to 30.

In general, the weight distribution of depth-wise convolution is very different for each channel compared to point-wise convolution in depth-wise separable convolution blocks, because there are cases where the weight variance of some channels is very large.

In INT8 quantization, this problem can be alleviated by adjusting each channel to have a different quantization range (per-channel granularity).

FP8 quantization can have different quantization ranges for each channel, like integer quantization. However, due to the characteristics of FP8 quantization, the expression range is limited to a range defined in advance by a formula by an additional _bias that can be adjusted by the user, and it is non-uniform quantization in which the quantization error is different for each certain range, so it is not suitable for data with large variance.

The purpose of the present invention is to minimize performance degradation that occurs when applying floating point format quantization of 8 bits or less or 16 bits or less to a deep learning model including depth-wise convolution.

The present invention provides a data-type-aware weight scaling device and method for floating-point quantization that can minimize accuracy loss due to floating-point quantization.

A weight scaling device for floating-point quantization according to one embodiment of the present invention comprises a scaler calculation unit that recognizes a data type and calculates a weight scaler value that adjusts the scale of a weight, and a multiplication unit that multiplies the weight by the calculated weight scaler.

In one embodiment, the multiplication unit includes a first multiplier that multiplies a weight of the depth-wise convolution by the inverse of a weight scaler, a second multiplier that multiplies a bias applied to the depth-wise convolution by the inverse of the weight scaler, and a third multiplier that multiplies a weight of the point-wise convolution by the weight scaler.

In one embodiment, the scaler calculation unit determines the ratio of the standard deviation of the weights of the corresponding layer and the standard deviation of the weights of the next layer of the corresponding layer as the weight scaler value. If the scaler value is less than 1, the scaler value is determined as 1.

In one embodiment, the weight scaling device can additionally reduce quantization error by applying an additional bias to a bias value used for point-wise convolution, and setting the value to a value at which the mean square error can be minimized when quantization is applied.

A weight scaling method for floating-point quantization according to one embodiment of the present invention comprises a weight scaler calculation step of recognizing a data type and calculating a weight scaler value that adjusts the scale of a weight, and a scaling step of multiplying the weight by the calculated weight scaler.

In one embodiment, the scaling step may be to multiply the weights of the depth-wise convolution by the inverse of the weight scaler, multiply the bias applied to the depth-wise convolution by the inverse of the weight scaler, and multiply the weights of the point-wise convolution by the weight scaler.

In one embodiment, the scaler calculation step determines the ratio of the standard deviation of the weights of the corresponding layer and the standard deviation of the weights of the next layer of the corresponding layer as the weight scaler value. If the scaler value is less than 1, the scaler value is determined as 1.

In one embodiment, the weight scaling method for floating-point quantization according to one embodiment of the present invention may further include a step of applying an additional bias to a bias value used for point-wise convolution. The additional bias is selected as a value at which a mean square error can be minimized when applying quantization.

According to the present invention, the variance of point-wise convolution weights having low sensitivity to quantization is adjusted slightly higher, and the variance of depth-wise convolution weights having high sensitivity to quantization is lowered, thereby minimizing the overall accuracy loss due to floating point quantization of 16 bits or less or 8 bits or less.

Figure 1 is a diagram for comparing the effect according to the quantization interval in the case of floating-point quantization in the case of low variance distribution and floating-point quantization in the case of high variance distribution.

Figure 2 is a diagram showing an exemplary data distribution according to dispersion.

Figure 3 shows an example of floating-point quantization error values according to dispersion.

Figure 4 is a block diagram showing a conventional configuration for applying a weight quantizer to successive depth-wise convolutions and point-wise convolutions.

FIG. 5 is a block diagram showing a configuration for applying a weight quantizer to successive depth-wise convolutions and point-wise convolutions by scaling weights according to one embodiment of the present invention.

Figure 6 is a table comparing the quantization error reduction effect when the conventional method and the method of the present invention are applied to MobileNet V1.

Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When adding reference numerals to components of each drawing, it should be noted that the same numerals are used for identical components as much as possible even if they are shown in different drawings. In addition, when describing the present disclosure, if it is determined that a specific description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b), etc. may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to 'include' or 'provide' a component, this does not mean that other components are excluded, but rather that other components can be further included, unless explicitly stated otherwise. In addition, terms such as 'part', 'module', etc. described in the specification mean a unit that processes at least one function or operation, and this can be implemented by hardware, software, or a combination of hardware and software.

The following description of the invention, together with the accompanying drawings, is intended to explain exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced.

In the following description, FP8 (1-4-3) (hereinafter referred to as FP8) quantization is used as an example for convenience of explanation, but the present invention is not limited thereto and can be applied to quantization of 16 bits or less, for example.

Unlike integer quantization that has the same quantization interval within the quantization range, FP8 quantization has both ranges that have the same quantization interval and ranges that have different quantization intervals due to the characteristics of floating points, as illustrated in Fig. 1. And due to this characteristic, the quantization error tends to increase as the absolute value of data increases. That is, in the graph of Fig. 1, the quantization interval of the part with large data values (the part far from 0 on the graph's x-axis) (indicated by the dotted box) is spaced further apart than the quantization interval of the part with small data values (the part close to 0 on the graph's x-axis), so the quantization error also increases. On the other hand, in the case of a low-variance distribution like Fig. 1 (a), since most of the data is concentrated in an area with dense intervals, the quantization error is less than in the case of a high-variance distribution like Fig. 1 (b).

Figure 2 shows the data distribution according to various variance values, and Figure 3 shows the quantization error values at that time. (a) is the case where variance = 0.5, (b) is the case where variance = 1, (c) is the case where variance = 2, and (d) is the case where variance = 4. As can be seen in Figure 3, the smaller the variance value, the smaller the quantization error, and the larger the variance value, the larger the quantization error.

The FP8(1-4-3) data type is predefined by the user. According to _{the default} bias , the expressible range [v _min , V _Max ] is determined by mathematical expression 1. Hereinafter, v _min is called the "minimum expressible value" and V _Max is called the "maximum expressible value". However, depending on the user's implementation, an additional bias (bias _extra ) can be added to adjust the expressible range within the candidates determined by mathematical expression 2. In this case, the optimal expressible range among the candidates can be selected according to the distribution range of the input data, thereby additionally reducing the quantization error. Here, the quantization error includes the rounding error and the truncation error. However, there is a limit to reducing the quantization error with this method under special conditions.

When applying FP8 quantization to a deep learning model that includes depth-wise convolution, the performance degradation occurs significantly compared to a deep learning model that does not include depth-wise convolution. In general, the weight distribution of depth-wise convolution has a characteristic of having a large variance compared to the subsequent point-wise convolution. This characteristic is reported to cause a large quantization error of the corresponding model when applying FP8 quantization to the weights of depth-wise convolution, which causes a large performance (e.g. accuracy) degradation of the deep learning model.

If the additional bias (bias _extra ) is increased to solve this problem, the minimum representation value (v _min ) that can be expressed becomes smaller, so values with small absolute values near 0 can be expressed better, but on the other hand, the maximum representation value (V _Max ) becomes smaller, so all weights with large absolute values are truncated to the maximum representation value, resulting in a large approximation error (truncation error).

On the other hand, if the additional bias is made small, the maximum expressible value (V _Max ) increases, but the rounding error increases accordingly. In addition, the absolute value of the minimum expressible value (v _min ) also increases, so data near 0 are truncated to the minimum expressible value (v _min ), which increases the approximation error (truncation error).

Ultimately, when applying FP8 quantization to data with large variance, the method of controlling additional bias is less effective.

In the present invention, an appropriate weight scaler is applied to the weights of depth-based convolution to make the weight dispersion smaller than before. This reduces the FP8 quantization error, thereby preserving the performance of the deep learning model to the maximum extent even after FP8 quantization is applied to the deep learning model. At this time, the inverse of the applied weight scaler is applied to the weights of the subsequent point-wise convolution within the depth-based separate convolution block to ensure that mathematically identical outputs are produced.

In the following description, we assume a pair of depth-wise convolution and point-wise convolution of a simple depth-wise separable convolution block. We also assume that the FP8 quantization of the weights does not allow additional bias adjustment on a per-channel basis. Basically, FP8 quantization is applied only on a per-tensor basis, but the present invention can be applied even if it varies on a per-channel basis.

The conventional FP8 quantization method applies an FP8 quantizer (40, 50) to each of the successive depth-wise convolutions (10) and point-wise convolutions (30), as illustrated in FIG. 4, to quantize the weights into an FP8 data type. The example of FIG. 1 shows a case where the i-th weight (W _i ^l ) (60) of the l-th layer and the j-th weight (W _j ^l+1 ) (80) of the l+1-th layer are quantized into an FP8 data type. b ^l represents the bias (70) of the l-th layer, and b ^l+1 represents the bias (90) of the l+1-th layer.

In the present invention, a scaler is applied to weights and biases as illustrated in FIG. 5. To this end, the weight scaling device (160) of the present invention comprises a scaler calculation unit (161) for recognizing a data type and calculating a weight scaler S that adjusts the scale of the weights, and multipliers (162, 163, 164) for multiplying the calculated weight scaler by the weights. The first multiplier (161) multiplies the weight of the depth-specific convolution (110) by the reciprocal of the weight scaler S (1/S), the second multiplier (162) multiplies the bias applied to the depth-specific convolution (110) by the reciprocal of the weight scaler (1/S), and the third multiplier (163) multiplies the weight of the point-specific convolution (130) by the weight scaler S.

In this way, an appropriate weight scaler is applied to the weights of the depth-based convolution to make them smaller than the variance of the original weights. This configuration reduces the FP8 quantization error, so that the performance of the deep learning model can be preserved to the maximum extent even after applying FP8 quantization to the model.

Here, we explain the effect of applying a weight scaler. Each layer of a deep learning model has weights (W) and input activation (A). f(·) is a nonlinear function that follows the layer.

의 모양을 갖는다. 이때

는 각각 합성곱 레이어 가중치의 높이(h)와 너비(w) 이다.The output of the lth layer can be expressed as in mathematical expression 3, and the output of the l +1th layer can be expressed as in mathematical expression 4. If the layer is a convolution layer, its weight is

It has the shape of . At this time

are the height (h) and width (w) of the convolution layer weights, respectively.

In a typical deep learning model, the lth layer and the l +1th layer are continuous, so A ^l+1 = O ^l . Therefore, if Equation 3 is substituted into Equation 4, it can be expressed as Equation 5.

At this time, if f(·) is a piece-wise linear function (e.g. ReLU, Leaky ReLU), then f ( ax ) = af ( x ), so Equation 5 can be expressed as Equation 6. In Equation 6, W' and b' represent the weight and bias after applying the weight scaler, respectively.

At this time, the variance of the new weight (W') ^l of the depth-wise convolution decreases as in mathematical expression 7.

As described above, in the case of FP8 quantization, the smaller the data size, the smaller the quantization error tends to be. Therefore, according to the present invention, the weight distribution is reduced, so that a relatively smaller quantization error can be achieved than the FP8 quantization error of the conventional depth-based convolution.

In the present invention, a method for determining a weight scaler S∈R ⁱ is described. If the weight scaler S _i of the i-th channel of the depth-wise convolution weights is determined to be too large, there is a risk that the variance of the point-wise convolution weight distribution will become too large.

Therefore, the ith weight scaler (S _i ^l ) of the lth layer is determined by the ratio of the standard deviation of W _i ^l and the standard deviation of W ^l+1 . If this scaler value is less than 1, the scaler value is determined as 1. At this time, the value of the scaler can be adjusted by the k _i value.

The method of determining the weight scaler (S _i ^l ) of the i-th channel can be expressed in a formula as shown in Mathematical Formula 8.

k _i is a factor that controls the effect of the scaler and can be determined in various ways, as shown in the following formula. For example, it can be set as the ratio of the exponent ( _lv ⁾ of 2 that is smaller than the absolute value of _Wi ^l and has the smallest difference from the absolute value of _Wi ^{l and the exponent (lv) of 2 that is smaller than the absolute value of Wi l+1} _and ^has the smallest difference from the absolute value of Wi _l +1, as shown in the following mathematical formula 9. Experimentally, it was found that the effect was most likely to be good when k i = 1.

This method can increase the variance of point-wise convolution by multiplying the inverse of the scaler to the point-wise convolution. However, it is generally reported that point-wise convolution weights have significantly smaller variance than depth-wise convolution weights and are robust to quantization errors because the number of weights is much larger. Therefore, the gain when applying a scaler to depth-wise convolution weights is greater than the loss when applying the inverse of the weight scaler to point-wise convolution weights.

However, since the inverse of the weight scaler is multiplied, the maximum value of the point-wise convolution weight may be greater than the maximum value of FP8 with extra_bias=4. In one embodiment, considering this, an additional bias may be applied to the bias value used for point-wise convolution, and the optimal value of the additional bias may be found and adjusted as in Equation 10.

That is, by selecting an additional bias that can minimize the mean square error when applying quantization, the loss when applying the inverse of the weight scaler to the point-by-point convolution weights can be reduced.

In this way, the variance of the point-wise convolution weights, which are less sensitive to quantization, can be adjusted slightly higher, and the variance of the depth-wise convolution weights, which are more sensitive to quantization, can be lowered, thereby minimizing the accuracy loss due to the overall FP8 quantization. Since the method of the present invention is based on the characteristics of the complex quantization interval of the FP data type itself, it will be applicable to all floating-point formats of 16 bits or less.

Fig. 6 is a table comparing the quantization errors when the conventional method and the method of the present invention are applied to MobileNet V1. As can be seen in Fig. 6, the FP8 format (FP8-WS) to which the method of the present invention is applied shows a performance improvement of +0.694 when the additional bias = 0 and +0.752 when the additional bias = 4, compared to the FP8 format (FP8) to which the method of the present invention is not applied.

Each component of the device or method according to the present invention may be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component may be implemented as software, and a microprocessor may be implemented to execute the function of the software corresponding to each component.

Various implementations of the systems and techniques described herein can be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable medium."

A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium can be a non-volatile or non-transitory medium, such as a ROM, a CD-ROM, a magnetic tape, a floppy disk, a memory card, a hard disk, a magneto-optical disk, a storage device, and may further include a transitory medium, such as a data transmission medium. In addition, the computer-readable recording medium can be distributed over a network-connected computer system, so that the computer-readable code can be stored and executed in a distributed manner.

Although the flowchart of this specification describes each process as being executed sequentially, this is only an illustrative description of the technical idea of one embodiment of the present disclosure. In other words, a person having ordinary skill in the art to which one embodiment of the present disclosure belongs may change the order described in the flowchart of this specification without departing from the essential characteristics of one embodiment of the present disclosure, or may modify and modify and apply various modifications and variations such as executing one or more of the processes in parallel. Therefore, the flowchart of this specification is not limited to a chronological order.

The above description is merely an example of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and variations may be made without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present embodiment.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority to Korean patent application No. 10-2023-0145071, filed in Korea on October 26, 2023, and Korean patent application No. 10-2023-0182302, filed in Korea on December 14, 2023, which are incorporated herein by reference in their entirety.

Claims (8)

A scaler calculation unit that calculates a weight scaler value that adjusts the scale of the weight by recognizing the data type, A multiplication part that multiplies the weights by the computed weight scaler. A weight scaling unit for floating point quantization having a . In the first paragraph, the multiplication unit, A first multiplier that multiplies the weights of the depth-wise convolution by the inverse of the weight scaler, A second multiplier that multiplies the inverse of the weight scaler by the bias applied to the depth-wise convolution, A third multiplier that multiplies the weights of the pointwise convolution by a weight scaler. A weight scaling unit for floating point quantization, comprising: In paragraph 1 or 2, The above scaler calculation unit determines the ratio of the standard deviation value of the weight of the corresponding layer and the standard deviation value of the weight of the next layer of the corresponding layer as the weight scaler value. Weight scaling unit for floating point quantization. In the third paragraph, the weight scaling device, Apply an additional bias to the bias value used for point-wise convolution, but select a value that can minimize the mean square error when applying quantization. Weight scaling unit for floating point quantization. A weight scaler calculation step that calculates a weight scaler value that recognizes the data type and adjusts the scale of the weight, Scaling step that multiplies the weights by the computed weight scaler. A weight scaling method for floating point quantization having. In the fifth paragraph, the scaling step, Multiply the weights of the depth-wise convolution by the inverse of the weight scaler, The bias applied to the depth-wise convolution is multiplied by the inverse of the weight scaler. Multiplying the weights of the pointwise convolution by a weight scaler, Weight scaling unit for floating point quantization. In clause 5 or 6, The above scaler calculation step determines the ratio of the standard deviation value of the weight of the corresponding layer and the standard deviation value of the weight of the next layer of the corresponding layer as the weight scaler value. Weight scaling unit for floating point quantization. In Article 7, It further provides a step of applying an additional bias to the bias value used for point-wise convolution. The above additional bias is selected to a value that can minimize the mean square error when applying quantization. Weight scaling unit for floating point quantization.

Images