KR102746707B1 - Circuit for implementing simplified sigmoid function and neuromorphic processor including the circuit - Google Patents

Abstract

The present disclosure relates to a simplified sigmoid function circuit and a neuromorphic processor including the same. The simplified sigmoid function circuit according to the present disclosure includes: a first circuit which performs an operation based on a simplified sigmoid function on input data when a sign of the input data in a real domain is positive; a second circuit which performs an operation based on the simplified sigmoid function on the input data when the sign of the input data in a real domain is negative; and a first multiplexer which selects and outputs one of an output of the first circuit and an output of the second circuit based on the sign of the input data, wherein the simplified sigmoid function is derived by converting a sigmoid function in a real domain into a sigmoid function in a log domain and by variationally transforming the sigmoid function in the log domain.

Description

{CIRCUIT FOR IMPLEMENTING SIMPLIFIED SIGMOID FUNCTION AND NEUROMORPHIC PROCESSOR INCLUDING THE CIRCUIT}

The present disclosure relates to artificial intelligence (AI) technology, and more specifically, to a simplified sigmoid function circuit used in an artificial neural network (ANN) and a neuromorphic processor including the same.

Recently, interest in artificial intelligence (AI), a core technology of the 4th industrial revolution, has been increasing. AI is the artificial implementation of human intelligence in machines, systems, etc., and can be implemented based on a learning algorithm called an artificial neural network (ANN). An artificial neural network is a statistical network that processes data in a similar way to a biological neural network. An artificial neural network can be used in various fields such as character recognition, image recognition, voice recognition, and face recognition.

As technology advances, the complexity of artificial neural networks increases, and the amount and type of data handled by artificial neural networks increases dramatically, leading to a rapid increase in the amount of computation required by artificial neural networks. In order to solve the problem that the data processing speed of artificial neural networks decreases significantly due to the increase in computational amount, various methods such as tensor decomposition, network pruning, and quantization have been proposed to reduce the amount of computation, but it is difficult to drastically reduce the amount of computation.

The present disclosure aims to provide a simplified sigmoid function circuit used in an artificial neural network (ANN) and a neuromorphic processor including the same.

A simplified sigmoid function circuit according to an embodiment of the present disclosure includes a first circuit which performs an operation based on a simplified sigmoid function on input data when a sign of the input data in a real domain is positive, a second circuit which performs an operation based on the simplified sigmoid function on the input data when the sign of the input data in a real domain is negative, and a first multiplexer which selects and outputs one of an output of the first circuit and an output of the second circuit based on the sign of the input data, wherein the simplified sigmoid function is derived by converting a sigmoid function in a real domain into a sigmoid function in a log domain and by variationally transforming the sigmoid function in the log domain.

For example, the first circuit includes a second multiplexer for selecting a first coefficient for the variational transform and a third multiplexer for selecting a second coefficient for the variational transform, and the second circuit includes a fourth multiplexer for selecting a third coefficient for the variational transform and a fifth multiplexer for selecting a fourth coefficient for the variational transform.

For example, the first circuit further includes a first multiplier which performs a product operation of the size of the input data and the first coefficient, and a first adder which performs a sum operation of the result of the product of the size of the input data and the first coefficient and the second coefficient, and the second circuit further includes a second multiplier which performs a product operation of the size of the input data and the third coefficient, and a second adder which performs a sum operation of the result of the product operation of the size of the input data and the third coefficient and the fourth coefficient.

As an example, the above variational transformation derives an approximated result through interval-wise variational transformation of the input data.

A neuromorphic processor according to an embodiment of the present disclosure includes an artificial neuron implementation element array including a plurality of artificial neuron implementation elements for performing an operation of an artificial neural network, wherein each of the plurality of artificial neuron implementation elements includes a summation circuit for performing a summation operation on a product of input data and a weight, and an activation function circuit for deriving an activation process through an activation function from a result of the operation of the summation circuit, wherein the activation function is derived by converting a sigmoid function of a real number domain into a sigmoid function of a log domain, and by variationally transforming the sigmoid function of the log domain.

For example, the activation function circuit includes at least one simplified sigmoid function circuit, wherein the at least one simplified sigmoid function circuit includes a first circuit which performs an operation based on the simplified sigmoid function on input data when the sign of the input data in the real domain is positive, a second circuit which performs an operation based on the simplified sigmoid function on the input data when the sign of the input data in the real domain is negative, and a first multiplexer which selects and outputs one of an output of the first circuit and an output of the second circuit based on the sign of the input data.

For example, the first circuit further includes a first multiplier which performs a product operation of the size of the input data and the first coefficient, and a first adder which performs a sum operation of the result of the product of the size of the input data and the first coefficient and the second coefficient, and the second circuit further includes a second multiplier which performs a product operation of the size of the input data and the third coefficient, and a second adder which performs a sum operation of the result of the product operation of the size of the input data and the third coefficient and the fourth coefficient.

For example, the first circuit includes a second multiplexer for selecting a first coefficient for the variational transform and a third multiplexer for selecting a second coefficient for the variational transform, and the second circuit includes a fourth multiplexer for selecting a third coefficient for the variational transform and a fifth multiplexer for selecting a fourth coefficient for the variational transform.

As an example, the above variational transformation derives an approximated result through interval-wise variational transformation of the input data.

For example, a neuromorphic processor according to the present disclosure further includes an input/output unit which receives input data from the outside and outputs a computation result of the artificial neural network for the input data to the outside, a control logic unit which receives the input data from the input/output unit and transmits the input data, a word line bias unit which transmits the input data from the control logic unit to the artificial neuron implementation element array, and a bit line bias and detection unit which detects a computation result for the input data from the artificial neuron implementation element array.

For example, a neuromorphic processor according to the present disclosure further includes a nonvolatile memory that stores information about a connection relationship between a plurality of artificial neuron implementation elements included in the artificial neuron implementation element array, and a volatile memory that stores the operation result detected from the artificial neuron implementation element array.

According to the simplified sigmoid function circuit and the neuromorphic processor including the same according to the present disclosure, the amount of computation of an artificial neural network can be drastically reduced by processing the sigmoid function, which is calculated in the real number domain, in the log domain.

FIG. 1 is a diagram illustrating an artificial neural network according to an embodiment of the present disclosure.
FIG. 2 is a diagram showing an artificial neuron according to an embodiment of the present disclosure.
FIG. 3 is a diagram showing a simplified sigmoid function circuit according to an embodiment of the present disclosure.
FIG. 4 is a diagram showing a neuromorphic processor to which a simplified sigmoid function circuit according to an embodiment of the present disclosure is applied.

Hereinafter, embodiments of the present disclosure will be described clearly and in detail to such an extent that a person skilled in the art can easily practice the present disclosure.

The terminology used herein is for the purpose of describing embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “singular” and “comprising” include plural forms unless the context clearly dictates otherwise. As used herein, the terms “comprises” and/or “comprising” do not exclude the presence or addition of one or more other components, steps, operations, and/or elements to the components, steps, operations, and/or elements stated.

The terms "first and/or second" used in this specification may be used to describe various components, but are only used for the purpose of distinguishing one component from another component, and are not intended to limit the component referred to by the terms. For example, without departing from the scope of the present disclosure, the first component may be referred to as the second component, and the second component may also be referred to as the first component.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in a meaning that can be commonly understood by a person of ordinary skill in the art to which this disclosure belongs. In addition, terms defined in commonly used dictionaries shall not be ideally or excessively interpreted unless explicitly specifically defined. In this specification, the same reference numerals may refer to the same components throughout the specification.

FIG. 1 is a diagram illustrating an artificial neural network (ANN, 10) according to an embodiment of the present disclosure. Referring to FIG. 1, the artificial neural network according to an embodiment of the present disclosure may include an input layer (IL), a hidden layer (HL), and an output layer (OL). The input layer (IL), the hidden layer (HL), and the output layer (OL) may be connected to each other through synapses (SN).

An artificial neural network (10) may include a plurality of artificial neurons (100). The plurality of artificial neurons (100) may include a plurality of input neurons that receive input data (X ₁ , X ₂ , ..., X _n ) from the outside, a plurality of hidden neurons that receive data from the plurality of input neurons and process the data, and a plurality of output neurons that receive data from the plurality of hidden neurons and generate output data (Y ₁ , Y ₂ , ..., Y _m ). An input layer (IL) may include a plurality of input neurons, a hidden layer (HL) may include a plurality of hidden neurons, and an output layer (OL) may include a plurality of output neurons.

The number of artificial neurons included in each of the input layer (IL), the hidden layer (HL), and the output layer (OL) is not limited to that shown in Fig. 1. In addition, the hidden layer (HL) may include more layers than that shown in Fig. 1. The number of hidden layers (HL) may be related to the accuracy and learning speed of the artificial neural network (10). In addition, the input data (X ₁ , X ₂ , …, X _n ) and the output data (Y ₁ , Y ₂ , …, Y _m ) may be various forms of data such as text and images.

FIG. 2 is a drawing showing an artificial neuron (100) according to an embodiment of the present disclosure. Referring to FIG. 2, the artificial neuron (100) may include a summation circuit (110) and an activation function circuit (120).

The summing circuit (110) can sum the input signals (A _{1 , A 2 , …, A K ) using the weights (W 1 , W 2 , …, W K ) . Each} of the input signals (A ₁ , A ₂ , …, A _K ) can be an output signal generated from an arbitrary artificial neuron. Each of the weights (W ₁ , W ₂ , …, W _K ) can represent the strength of a synapse (SN, see FIG. 1), in other words, the degree of coupling between one artificial neuron and another artificial neuron. The summing circuit (110) can derive a summing result (B) by multiplying the weights (W ₁ , W ₂ , …, W _K ) by the input signals (A ₁ , A ₂ , …, A _K ). The summing result (B) can be expressed by mathematical expression 1.

The activation function circuit (120) can derive an activation result (C) by using the sum result (B) and the activation function (Activation Function, f). The activation function (f) according to the embodiment of the present disclosure can be a simplified sigmoid function. The sigmoid function can be used to derive a nonlinear value in a linear multiperceptron, and may include a logistic function as an example. The sigmoid function can be defined in a real number domain and can be expressed by mathematical expression 2. In mathematical expression 2, sig(x) represents a sigmoid function in a real number domain, and x represents a real variable.

In the embodiments of the present disclosure, the simplified sigmoid function used as the activation function (f) may be a variational form of the sigmoid function in the log domain in order to reduce the amount of computation of the artificial neural network. That is, the activation function (f) used in the embodiments of the present disclosure may be derived through a process of deriving a sigmoid function in the log domain and variationally transforming the derived sigmoid function in the log domain.

The sigmoid function in the log domain can be derived by taking the natural logarithm of both sides of the sigmoid function in the real domain presented in Equation 2. The process of deriving the sigmoid function in the log domain can be expressed as processes (1) to (4) of Equation 3. In Equation 3, sig(x) represents the sigmoid function in the real domain, SIG(X) represents the sigmoid function in the log domain, x represents a real variable, and X represents a variable in the log domain. Hereinafter, for the sake of clarity of notation, e ^x is expressed in the form of exp(x).

The sigmoid function of the log domain derived from mathematical expression 3 can be variationally transformed to reduce the amount of computation of the artificial neural network. The variational transformation of the sigmoid function of the log domain can be derived differently depending on the sign of the real variable x. First, when the real variable x is positive, the derivation process can be expressed as processes (1) to (12) of mathematical expression 4. In mathematical expression 4, F(x) denotes an approximate expression derived through the variational transformation, and D(x) denotes the difference between the sigmoid function SIG(X) of the log domain and the approximate expression. In addition, Y denotes a function that minimizes D(x).

When the real variable x is negative, the sigmoid function in the log domain is expressed as in Equation 5. In Equation 5, since x is negative, it can be expressed as the product of xp, which is a positive number, and -1. In Equation 5, sig(x) means the sigmoid function in the real domain, and SIG(Xp) means the sigmoid function in the log domain.

When the real variable x is negative, the derivation process can be expressed as processes (1) to (12) of Equation 6. In Equation 6, F(x) denotes an approximate formula derived through variational transformation, and D(x) denotes the difference between the sigmoid function SIG(Xp) in the log domain and the approximate formula. In addition, Y denotes a function that minimizes D(xp).

The approximate formulas F(X) and F(Xp) derived from mathematical expressions 4 and 6 can be used as an activation function (f) to derive an activation result (C) in an activation function circuit (120).

FIG. 3 is a diagram showing a simplified sigmoid function circuit (200) according to an embodiment of the present disclosure. The simplified sigmoid function circuit (200) is a circuit for implementing an activation function (f) and may be included in an activation function circuit (120, see FIG. 2). Referring to FIG. 3, the simplified sigmoid function circuit (200) may include a first circuit (210), a second circuit (220), and a first multiplexer (230). The first circuit (210) may include a first comparator (211), a second multiplexer (212a), a third multiplexer (212b), a first multiplier (213), and a first adder (214). The second circuit (220) may include a second comparator (221), a fourth multiplexer (222a), a fifth multiplexer (222b), a second multiplier (223), and a second adder (224).

According to an embodiment of the present disclosure, an activation result (F) (corresponding to the activation result (C) disclosed in FIG. 2) can be derived through a simplified sigmoid function circuit (200). In the log domain, an input can be expressed as a vector with a direction sign Xs and a size X. The simplified sigmoid function circuit (200) is input If the direction sign of the input is positive (+), the output value from the first circuit (210) can be derived as an active result (F). When the direction sign of the second circuit (220) is negative (-), the output value from the second circuit (220) can be derived as an active result (F).

FIG. 4 is a diagram showing a neuromorphic processor (1000) to which a simplified sigmoid function circuit (200, see FIG. 3) according to an embodiment of the present disclosure is applied. Referring to FIG. 4, the neuromorphic processor (1000) may include an artificial neuron implementation element array (1100), a word line bias unit (1200), a bit line bias and detection unit (1300), a control logic unit (1400), a nonvolatile memory (1500), a volatile memory (1600), and an input/output unit (1700).

The artificial neuron implementation element array (1100) may be a hardware implementation of the artificial neural network (10, see FIG. 1) disclosed in the above-described FIG. 1. The artificial neuron implementation element array (1100) may include elements implementing a plurality of artificial neurons (100, see FIG. 1), and may have a structure in which the plurality of artificial neuron (100) implementation elements are arranged in rows and columns. Each of the plurality of artificial neuron (100) implementation elements may include a simplified sigmoid function circuit (200, see FIG. 3) disclosed in the above-described FIG. 3. The artificial neuron implementation element array (1100) may output a result based on the simplified sigmoid function. In FIG. 4, the word line (WL) and the bit line (BL) connected to the artificial neuron implementation element array (1100) are illustrated as one, but this is only to reduce the complexity of the drawing, and means the word line (WL) and the bit line (BL) connected to each of the plurality of artificial neuron (100) implementation elements included in the artificial neuron implementation element array (1100).

The word line bias unit (1200) can receive input data from the control logic unit (1400) and transmit the input data to each of a plurality of artificial neuron (100) implementation elements included in the artificial neuron implementation element array (1100) through the word line (WL). In addition, the word line bias unit (1200) can supply current for recording weights for a plurality of synapse (SN, see FIG. 1) connections included in the artificial neuron implementation element array (1100) through the word line (WL).

The bit line bias and detection unit (1300) can bias the bit line (BL) to a ground voltage when performing an artificial neural network operation on each of a plurality of artificial neuron (100) implementation elements included in the artificial neuron implementation element array (1100). In addition, the bit line bias and detection unit (1300) can obtain an operation result of a plurality of artificial neuron (100) implementation elements included in the artificial neuron implementation element array (1100) by detecting the amount of current through the bit line (BL).

The control logic unit (1400) can read information stored in the nonvolatile memory (1500) and control the word line bias unit (1200) and the bit line bias and detection unit (1300) based on the read information. In addition, the control logic unit (1400) can transfer an initial input received through the input/output unit (1700) as input data to the word line bias unit (1200) or store it in the volatile memory (1600). In addition, the control logic unit (1400) can transfer a result output from the artificial neuron implementation element array (1100) as input data to the word line bias unit (1200) or store it in the volatile memory (1600).

The nonvolatile memory (1500) can store information about the connection relationship between a plurality of artificial neuron (100) implementation elements included in the artificial neuron implementation element array (1100). That is, the nonvolatile memory (1500) can include information about the entire structure of the artificial neural network implemented by the neuromorphic processor (1000).

The volatile memory (1600) can store an initial input input from an input/output unit (1700) and a result output from an artificial neuron implementation element array (1100). The input/output unit (1700) can receive an initial input from the outside and transmit it to a control logic unit (1400), and can receive a result output from an artificial neuron implementation element array (1100) from the control logic unit (1400) and output it to the outside.

The above-described contents are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the above-described embodiments, but also embodiments that are simply designed or can be easily changed. In addition, the present disclosure will also include technologies that can be easily modified and implemented using the embodiments. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined by the claims described below as well as the equivalents of the claims of the present disclosure.

10: Artificial Neural Network (ANN)
100 : Artificial Neuron
110 : Summatation Circuit
120: Activation Function Circuit

Claims (11)

A first circuit which performs an operation based on a simplified sigmoid function corresponding to a first-order function for the input data when the sign in the real number domain of the input data is positive;
A second circuit that performs an operation based on a simplified sigmoid function corresponding to a second linear function for the input data when the sign in the real domain of the input data is negative; and
Including a first multiplexer that selects and outputs one of the outputs of the first circuit and the output of the second circuit based on the sign of the input data,
The above first and second linear functions are simplified sigmoid function circuits corresponding to the variational transformation of the sigmoid function of the log domain generated by transforming the sigmoid function of the real domain into the log domain. delete delete delete An artificial neuron implementation element array comprising a plurality of artificial neuron implementation elements for performing operations of an artificial neural network,
Each of the above multiple artificial neuron implementation elements:
A summing circuit that performs a sum operation on the product of input data and weights; and
It includes an activation function circuit that derives an activation result corresponding to a sigmoid function from the operation result of the above summation circuit,
A neuromorphic processor in which the above activation result is derived based on a linear function corresponding to a variational transform of a sigmoid function in a log domain generated by converting the sigmoid function in the real domain into a log domain.
delete delete delete delete delete delete