WO2025110327A1 - Calibration method and apparatus using input/output distribution of adjacent layer - Google Patents

Abstract

The present invention relates to a calibration method and apparatus using an input/output distribution of an adjacent layer. According to one aspect of the present invention, provided is a calibration method comprising: an operation process of generating a first operation result in an operation layer of an artificial neural network; a calibration process of acquiring a first effective input range by using information about another layer adjacent to the operation layer; a first quantization process of generating a first quantization result by quantizing the first operation result on the basis of the first effective input range; and an activation process of generating a first activation output by inputting the first quantization result to the first activation function.

Description

Calibration method and device utilizing input/output distribution of adjacent layers

The present disclosure relates to a calibration method and device utilizing input/output distribution of adjacent layers.

The content described below merely provides background information related to the present embodiment and does not constitute prior art.

Deep learning models consist of a combination of blocks consisting of layers (convolution layers, linear layers) that perform linear operations on learnable weights and activation functions. These blocks can be combined in dozens to hundreds of combinations.

Since the parameters and operations of deep learning models require precise representation, floating point data types such as FP32 and FP16 are generally used.

When using FP32 or FP16 as the data type, the size of FP32 is 32 bits and the size of FP16 is 16 bits, so a lot of memory is required for the operation.

Typically, in deep learning models, the computational results are quantized to accelerate computation and reduce memory usage, and expressed as data types with fewer bits (e.g., 8-bit INT8 and FP8 data types).

Quantization is mainly applied to weights, operation results, and activation. When quantizing the operation results and quantizing the activation, which is the output of the activation function, a threshold value is required to approximate and express the actual input distribution of the quantization, and this process is called the calibration process.

The calibration process refers to the step of analyzing the output distribution of a specific layer to determine an appropriate threshold, which is calculated and used when quantizing the operation results and quantizing the activation.

Conventional calibration uses the most efficient calibration method among the representative calibration methods, namely maximum, percentile, and entropy (max/percentile/entropy) calibration.

Figure 1a is a diagram illustrating the absolute value frequency distribution of the operation output value of a specific layer and the percentile position that serves as a calibration reference point, and Figure 1b is a diagram illustrating the quantization target area and the quantization exclusion area as a result of the calibration.

As illustrated in FIG. 1a, in order to perform symmetric quantization on an operation output value, when the value at the 99.99% percentile point of the absolute value frequency distribution of the operation output value is set as a calibration reference point, as illustrated in FIG. 1b, symmetric quantization is performed on the operation output between the + calibration reference point and the - calibration reference point corresponding to the values within the 99.99% percentile point to generate the first quantized output, while values whose absolute values of the operation output are larger than the calibration reference point (i.e., the area indicated by Clip in FIG. 1b) are excluded from the quantization target, thereby achieving relatively efficient quantization using a quantization code of a constant number of bits.

Figure 2 is a diagram showing the input area of the activation function along with the frequency distribution of the entire operation output value.

As illustrated in Fig. 2, when the valid input region of the activation function is defined as 0≤x≤6, when the first quantized output is input to the activation function, the first quantized output outside the valid input region of the activation function becomes an unnecessary input to the activation function, and an inefficiency occurs in which a quantization code is assigned to the first quantized output that is not unnecessary to the activation function.

Therefore, when performing calibration using a conventional method, when considering the quantization and activation processes after calibration, information loss occurs due to the allocation of unnecessary quantization codes when quantizing the calibration results. Therefore, in order to minimize this loss, calibration needs to be made more precise.

The main purpose of the present disclosure is to provide a calibration method and device utilizing the input/output distribution of adjacent layers.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to one aspect of the present disclosure, a calibration method is provided, including: a computational process for generating a first computational result in a computational layer of an artificial neural network; a calibration process for obtaining a first valid input range by using information of another layer adjacent to the computational layer; a first quantization process for quantizing the first computational result based on the first valid input range to generate a first quantized result; and an activation process for inputting the first quantized result to the first activation function to generate a first activation output.

A computer program stored on a computer-readable recording medium is provided to execute each process included in the above calibration method.

According to another aspect of the present disclosure, a calibration device is provided, including: a calculation unit that generates a first calculation result in a calculation layer of an artificial neural network; a calibration unit that obtains a first valid input range by using information of another layer adjacent to the calculation layer; a first quantization unit that quantizes the first calculation result based on the first valid input range to generate a first quantization result; and an activation unit that inputs the first quantization result to the first activation function to generate a first activation output.

According to an embodiment of the present disclosure, there is an effect of further elaborating the quantization process prior to activation in an artificial neural network.

When using an activation function other than the types of standard activation functions, it has the effect of effectively finding a valid input range.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1a is a diagram illustrating the absolute value frequency distribution of the operation output value of a specific layer and the percentile position that serves as a calibration reference point, and Figure 1b is a diagram illustrating the quantization target area and the quantization exclusion area as a calibration result.

Figure 2 is a diagram showing the input area of the activation function along with the frequency distribution of the entire operation output value.

FIG. 3 is a block diagram illustrating the configuration of a calibration device (300) according to one embodiment of the present disclosure.

Fig. 4a is a diagram illustrating the input and output of the activation function ReLU6(), Fig. 4b is a diagram illustrating the input and output of the activation function sigmoid(), and Fig. 4c is a diagram illustrating the input and output of the activation function Swish().

Figure 5 is a diagram illustrating a graph of the activation function Swish() and its derivative, the Swish'() function.

FIG. 6 is a flowchart illustrating a calibration method according to one embodiment of the present disclosure.

Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When adding reference numerals to components of each drawing, it should be noted that the same numerals are used for identical components as much as possible even if they are shown in different drawings. In addition, when describing the present disclosure, if it is determined that a specific description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b), etc. may be used. These symbols are only for distinguishing the components from other components, and the nature or order or sequence of the components is not limited by the symbols. When a part in the specification is said to "include" or "provide" a component, this does not mean that other components are excluded, but rather that other components can be further included, unless explicitly stated otherwise.

The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

FIG. 3 is a block diagram illustrating the configuration of a calibration device (300) according to one embodiment of the present disclosure.

As illustrated in FIG. 3, a calibration device (300) according to the present embodiment may be implemented by including a calculating unit (310), a calibrating unit (320), a first quantizing unit (330), an activating unit (340), a second quantizing unit (350), and a table storing unit (360). The calibration device (300) according to the present embodiment may be implemented by omitting some of the components of FIG. 3 or by adding other components not illustrated in FIG. 3.

The calculating unit (310) generates a first calculation result in a calculation layer during deep learning. For example, the calculating unit (310) performs a calculation of one layer in an artificial neural network such as a convolutional neural network (CNN) and a recurrent neural network (RNN) related to deep learning.

For example, in the case of performing matrix multiplication using a kernel and a part of a data set of an input image in a convolutional neural network, as illustrated in FIG. 3, the operation unit (310) performs matrix multiplication ∑w i x i using the weights (w ₁ , w ₂ , w ₃ , w ₄ ) of the corresponding kernels for each of the four data _sets (x ₁ , x ₂ , x ₃ , x ₄ ) of the input _image , thereby outputting the first operation result.

The operation unit (310) applies kernel weights to each of a plurality of different data sets, performs matrix multiplication for each, and outputs each first operation result.

The operation of the calibration unit (320) is performed each time a matrix multiplication is performed by the operation unit (310) and the first operation result is output.

If the output distribution of an activation function can be predicted in a block consisting of a layer (Convolution Layer, Linear layer) that performs linear operations using weights and an activation function, this output distribution can be utilized in the calibration stage of the adjacent previous layer.

If the characteristic information of the activation function as well as the output distribution of the activation function for the actual input is utilized in the previous layer, the quantization process of the previous layer can be performed more precisely.

The calibration unit (320) obtains a first valid input range related to the relationship between the input value of the first activation function and the output value of the first activation function related to the first operation result of the operation unit (110).

Fig. 4a is a diagram illustrating the input and output of the activation function ReLU6(), Fig. 4b is a diagram illustrating the input and output of the activation function sigmoid(), and Fig. 4c is a diagram illustrating the input and output of the activation function Swish().

It is assumed that the calibration unit (320) knows what the first activation function related to the first operation result is.

As shown in Fig. 4a, if the first activation function related to the first operation result is ReLU6(x), then y=ReLU6(x) is the following formula:

If x<0, then y=0

If 0≤x≤6, then y=x

If x>60, then y=6

is defined as

Therefore, the calibration unit (320) obtains a valid input range [0:6] of ReLU6(x), where [0:6] is defined as a value greater than or equal to 0 and less than or equal to 6.

In addition, as illustrated in FIG. 4b, when the first activation function related to the first operation result is sigmoid(z), in y=sigmoid(z)=1/(1+e ^-z ), if z<-6 or z>6, y≒0. Accordingly, the calibration unit (320) obtains the valid input range [-6:6] of the first activation function sigmoid(z), and the valid input range [-6:6] is defined as a value greater than or equal to -6 and less than or equal to 6.

In addition, as illustrated in Fig. 4c, when the first activation function related to the result of the first operation of the activation function is Swish(x), in y=Swish(x)=x*sigmoid(x), if x<-6, y≒0. Accordingly, the calibration unit (320) obtains the valid input range [-6:INF] of the first activation function Swish(x), and the valid input range [-6:INF] is defined as a value greater than or equal to -6.

The table storage unit (360) stores the valid input range of each activation function for at least one activation function in the valid range table (370).

In the present embodiment, when one of ReLU6(), sigmoid(), and Swish() is used as an activation function, the table storage unit (360) stores the valid input ranges for each of the multiple candidate activation functions ReLU6(), sigmoid(), and Swish() in the valid range table (370).

Since the calibration unit (320) knows which function is the first activation function related to the first operation result, the calibration unit (320) searches for the first activation function among the multiple candidate activation functions within the valid range table (370).

If the first activation function related to the first operation result exists in the valid range table (370), the calibration unit (320) obtains the valid input range related to the found first activation function from the valid range table (370) and sets the obtained valid input range as the first valid input range.

The first quantization unit (330) quantizes the first operation result based on the first valid input range to generate the first quantization result. The first quantization unit (330) does not perform quantization on the first operation result that is outside the first valid input range.

Since the first operation result is quantized within the first valid input range, a finer quantization is possible compared to cases where it is calibrated and quantized using methods such as maximum, percentile, and entropy, so there is a relatively high possibility that data loss due to quantization of the first operation result will be minimized.

There may also be cases where the first activation function related to the first operation result does not exist within the valid range table (370).

If a user is given a first activation function that does not exist in the valid range table (370), the calibration unit (320) obtains the first derivative of the output value with respect to the input value on the given first activation function, and obtains the first valid input range from the obtained first derivative.

The first valid input range of the first activation function can utilize the first derivative of the first activation function.

When the first derivative of the first activation function has a value of 0 continuously for a certain interval of input values or the difference from 0 is less than or equal to a preset size continuously for a certain interval of input values, the first valid input range of the first activation function is obtained based on the certain interval.

Figure 5 is a diagram illustrating a graph of the activation function Swish() and its derivative, the Swish'() function.

If information on the valid input range of the activation function Swish() does not exist in the valid range table (370) and Swish() is given as the first activation function, the calibration unit (320) obtains the first valid input range of the first activation function based on the first derivative value of Swish() as the given first activation function.

As illustrated in Fig. 5, in the first activation function Swish(), in the input interval -6 or less, cases in which the difference between the first derivative of Swish() and 0 is less than or equal to a preset size appear continuously for a certain interval, and in the input interval -6 or greater, cases in which the difference between the first derivative of Swish() and 0 is less than or equal to a preset size do not appear continuously for a certain interval.

Accordingly, in the case of Fig. 5, the calibration unit (320) sets the section of [-6:INF], excluding the section in which the difference between the first derivative value of Swish() and 0 is less than or equal to a preset size, as the first valid input range of the first activation function Swish().

If information on the valid input range of the activation function ReLU6() does not exist in the valid range table (370) and ReLU6() is given as the first activation function, the calibration unit (320) obtains the first valid input range of the first activation function based on the first derivative value of ReLU6() as the given first activation function.

In the case of ReLU6(x), we can see that the first derivative of ReLU6(x) continuously appears as 0 because the output value is constant when the input value x is less than or equal to 0 and when the input value x is greater than or equal to 6.

Accordingly, the calibration unit (320) sets the interval [0:6], excluding the interval x<0, x>6 where the first derivative of ReLU6() continuously appears as 0, as the first valid input range of the first activation function ReLU6().

When a first activation function that does not exist in the valid range table (370) is used, the calibration unit (320) obtains an expected lower limit value and an expected upper limit value related to the first operation result.

The calibration unit (320) obtains a certain section in which the first derivative has a value of 0 continuously for input values between an expected lower limit value and an expected upper limit value or in which the difference between the first derivative of the first activation function and 0 continuously is less than or equal to a preset size, and determines the remaining section, excluding the certain section, among the sections between the expected lower limit value and the expected upper limit value, as the first valid input range of the first activation function.

The calibration unit (320) obtains a first predetermined interval in which the first derivative of the first activation function in the interval from 0 to the expected lower limit value has a value of 0 continuously or the difference between the first derivative and 0 continuously is equal to or smaller than a preset size, obtains a second predetermined interval in which the first derivative of the first activation function in the interval from 0 to the expected upper limit value has a value of 0 continuously or the difference between the first derivative and 0 continuously is equal to or smaller than a preset size, and determines a range that is equal to or larger than the first predetermined interval and equal to or smaller than the second predetermined interval as a first valid input range.

In this way, when performing the process of obtaining the first and second schedule intervals, there is no need to check the first derivative for the first schedule interval or other smaller input values, and there is no need to check the first derivative for the second schedule interval or other larger input values, so that the first valid input range can be efficiently determined.

As described above, the first quantization unit (330) quantizes the first operation result based on the first valid input range to generate the first quantization result. For the first operation result outside the first valid input range, the first quantization unit (330) excludes the first operation result from the quantization target, thereby allowing the quantization result for the first operation result to be expressed more accurately.

The activation unit (150) inputs the first quantization result into the first activation function to generate a first activation output.

It can be seen that the quantization loss due to the first quantization result is minimized because the effective input range of the first activation function used by the activation unit (150) is identical to the output range of the first quantization result.

The second quantization unit (350) quantizes the first activation output to generate a second quantization result. During the second quantization, calibration is performed on the first activation output so that an effective quantization input range for the first activation output can be obtained.

When calibrating the first activation output, the most efficient method among the maximum, percentile and entropy methods can be selected.

FIG. 6 is a flowchart illustrating a calibration method according to one embodiment of the present disclosure.

A calibration method according to one embodiment of the present disclosure is performed by a calibration device (300).

The operation unit (310) performs an operation process to generate a first operation result in the operation layer of the artificial neural network (S610).

The calibration unit (320) performs a calibration process to obtain a first valid input range by using information from another layer adjacent to the operation layer (S620).

The first quantization unit (330) performs a first quantization process to generate a first quantization result by quantizing the first operation result based on the first valid input range (S630).

The activation unit (340) performs an activation process that inputs the first quantization result into the first activation function to generate a first activation output (S640).

Each component of the device or method according to the present invention may be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component may be implemented as software, and a microprocessor may be implemented to execute the function of the software corresponding to each component.

Various implementations of the systems and techniques described herein can be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable medium."

A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium can be a non-volatile or non-transitory medium, such as a ROM, a CD-ROM, a magnetic tape, a floppy disk, a memory card, a hard disk, a magneto-optical disk, a storage device, and may further include a transitory medium, such as a data transmission medium. In addition, the computer-readable recording medium can be distributed over a network-connected computer system, so that the computer-readable code can be stored and executed in a distributed manner.

Although the flowchart/timing diagram of this specification describes each process as being executed sequentially, this is only an illustrative description of the technical idea of one embodiment of the present disclosure. In other words, a person having ordinary skill in the art to which one embodiment of the present disclosure belongs may change and modify and apply various modifications and variations such as changing the order described in the flowchart/timing diagram and executing it or executing one or more of the processes in parallel without departing from the essential characteristics of one embodiment of the present disclosure. Therefore, the flowchart/timing diagram is not limited to a chronological order.

The above description is merely an illustrative description of the technical idea of the present embodiment, and those with ordinary skill in the art to which the present embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present embodiment.

[Explanation of symbols]

300: Calibration device 310: Operation unit

320: Calibration section 330: First quantization section

340: Activation section 350: Second quantization section

360: Table storage 370: Valid range table

Statement regarding sponsored research or development

The present invention is a result of a research project (Project Unique Number: 1711117060, Subproject Number: 2020-0-01305-001, Ministry: Ministry of Science and ICT, Project Management (Specialized) Agency: National IT Industry Promotion Agency, Research Project Name: Next-Generation Intelligent Semiconductor Technology Development (Design) R&D, Research Project Name: Development of 2,000 TFLOPS-Class Server Artificial Intelligence Deep Learning Processor and Module, Contribution Rate: 1/1, Project Performing Agency: SK Telecom Co., Ltd., Sapion Korea Co., Ltd., Research Period: 2020.04. 01. ~ 2027.12. 31.).

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority to Korean patent application No. 10-2023-0162768, filed on November 21, 2023, and Korean patent application No. 10-2023-0181018, filed on December 13, 2023, which are incorporated herein by reference in their entirety.

Claims (13)

A computational process that produces the first computational result in the computational layer of an artificial neural network; A calibration process for obtaining a first valid input range by using information from another layer adjacent to the above operation layer; A first quantization process that quantizes the first operation result based on the first valid input range to generate a first quantization result; and An activation process that inputs the first quantization result to the first activation function to generate a first activation output. A calibration method comprising: In the first paragraph, The above calibration process is, A calibration method characterized by obtaining the first valid input range related to the relationship between the input value of the first activation function and the output value of the first activation function for the first activation function of the activation layer related to the first operation result. In the second paragraph, The above activation layer is, A calibration method characterized in that the activation layer appears immediately after quantization of the first operation result. In the second paragraph, Further comprising a table storage process for storing the valid input range of each activation function in a valid range table for at least one activation function, The above calibration process is, A calibration method characterized by obtaining the first activation function among the at least one activation function within the above valid range table, and obtaining the valid input range related to the obtained first activation function as the first valid input range. In the second paragraph, The above calibration process is, A calibration method characterized by obtaining a first derivative of the output value with respect to the input value and obtaining the first valid input range from the first derivative. In paragraph 5, The above calibration process is, A calibration method characterized in that the first valid input range is obtained based on the predetermined interval when the first differential value has a value of 0 continuously for a certain interval of the input value or the difference from 0 continuously for a certain interval of the input value is less than or equal to a preset size. In Article 6, The above calibration process is, Obtain the expected lower bound value and the expected upper bound value related to the above first operation result, A calibration method characterized in that the first differential value continuously has a value of 0 for the input value between the expected lower limit value and the expected upper limit value, or the first differential value continuously has a difference from 0 less than or equal to a preset size, is obtained in a predetermined interval, and an interval remaining between the expected lower limit value and the expected upper limit value, excluding the predetermined interval, is determined as the first valid input range. In Article 6, The above calibration process is, Obtain the expected lower bound value and the expected upper bound value related to the above first operation result, A calibration method characterized in that a first predetermined interval is obtained in which the first derivative continuously has a value of 0 for the input values in the interval from 0 to the expected lower limit value or the difference between the first derivative and 0 is less than or equal to a preset size, a second predetermined interval is obtained in which the first derivative continuously has a value of 0 for the input values in the interval from 0 to the expected upper limit value or the difference between the first derivative and 0 is less than or equal to a preset size, and a range of values greater than or equal to the first predetermined interval and less than the second predetermined interval is determined as the first valid input range. A computer program stored on a computer-readable recording medium for executing each process included in a calibration method according to any one of claims 1 to 8. A computational unit that generates the first computational result in the computational layer of an artificial neural network; A calibration unit that obtains a first valid input range by using information of another layer adjacent to the above operation layer; A first quantization unit that quantizes the first operation result based on the first valid input range to generate a first quantization result; and An activation unit that inputs the first quantization result to the first activation function to generate a first activation output. A calibration device comprising: In Article 10, The above calibration unit, A calibration device characterized in that it obtains the first valid input range related to the relationship between the input value of the first activation function and the output value of the first activation function for the first activation function of the activation layer related to the first operation result. In Article 11, Further comprising a table storage unit storing the valid input range of each activation function in a valid range table for at least one activation function, The above calibration unit, A calibration device characterized in that it obtains the first activation function among the at least one activation function within the above valid range table, and obtains the valid input range related to the obtained first activation function as the first valid input range. In Article 11, The above calibration unit, A calibration device characterized in that it obtains the first derivative of the output value with respect to the input value and obtains the first valid input range from the first derivative.

Images