CN116720555A - Systems and methods for storage bits in artificial neural networks - Google Patents

Abstract

The present disclosure relates to systems and methods for memory bits in artificial neural networks. The present disclosure is particularly directed to a device, which includes: an input circuit; a weight operation circuit electrically connected to the input circuit; and a deviation operation circuit electrically connected to the weight operation circuit. ; a storage circuit, the storage circuit is electrically connected to the weight operation circuit and the deviation operation circuit; and an activation function circuit, the activation function circuit is electrically connected to the deviation operation circuit, wherein at least the weight operation circuit, The deviation operation circuit and the storage circuit are located on the same chip.

Description

Systems and methods for storage bits in artificial neural networks

Cross-references to related applications

This application claims the benefit of U.S. Provisional Patent Application No. 63/268,953, filed on March 7, 2022, the entire contents of which are incorporated herein by reference.

Technical field

Embodiments of the present disclosure relate particularly to storage bits. More specifically, certain embodiments of the present disclosure relate to memory bits in artificial neural networks.

introduction

An artificial neural network may have an input layer and an output layer with multiple hidden layers. Each layer after the input layer may have multiple hardware neurons that perform various operations. For example, each hardware neuron may perform a multiply and accumulate (MAC) operation on the input and weight values, sum the product of the MAC operation and any bias values, and/or perform an activation function, such as to generate the output layer. Rectified linear unit (ReLU) activation function or sigmoid function of the output value.

For some conventional hardware neurons, weight and bias values may require storage operations, retrieval operations, and/or modification operations in the context of these artificial neural networks. For example, in inference applications, the weight values and bias values of each hardware neuron may need to be stored in non-volatile memory off-chip. During the use of hardware neurons, weight and bias values can be loaded from off-chip non-volatile memory into on-chip random access memory (RAM) registers that implement artificial neural networks. Off-chip memory access to weight and bias values may significantly increase the chip's power consumption and/or increase the latency of hardware neuron operations. Therefore, the hardware neuron may need to be configured to reduce the power consumption and latency typically associated with loading these values from non-volatile memory to the hardware neuron.

Description of the drawings

During the following detailed description, reference will be made to the accompanying drawings. The drawings illustrate different aspects of the disclosure, and where appropriate, reference numbers showing the same structures, components, materials and/or elements in the different drawings are labeled similarly. It is understood that various combinations of structures, components and/or elements in addition to those specifically shown are contemplated and are within the scope of the present disclosure.

Additionally, numerous embodiments of the present disclosure are described and illustrated herein. The disclosure is neither limited to any single aspect or embodiment thereof, nor to any combination and/or permutation of these aspects and/or embodiments. Furthermore, each aspect of the disclosure and/or embodiments thereof may be used alone or in combination with one or more other aspects of the disclosure and/or embodiments thereof. For the sake of brevity, certain permutations and combinations are not individually discussed and/or shown herein; however, all permutations and combinations are considered to be within the scope of the invention.

Figure 1 depicts a functional diagram of an exemplary artificial neural network according to an exemplary embodiment of the present disclosure.

2 depicts an example of a first hardware neuron of the artificial neural network of FIG. 1 according to an exemplary embodiment of the present disclosure.

3 depicts an example of a second hardware neuron of the artificial neural network of FIG. 1 according to an exemplary embodiment of the present disclosure.

4 depicts an exemplary memory circuit configuration of a hardware neuron according to an exemplary embodiment of the present disclosure.

Figure 5 depicts various bridge element configurations of a hardware neuron's memory circuit in accordance with an exemplary embodiment of the present disclosure.

6A depicts an example of a circuit of a multiple-times programmable storage circuit of a hardware neuron configured for writing a first value in accordance with an exemplary embodiment of the present disclosure.

6B depicts an example of a circuit of a multiple-times programmable storage circuit of a hardware neuron configured for writing a second value in accordance with an exemplary embodiment of the present disclosure.

7A depicts an example of a circuit of a one-time programmable memory circuit of a hardware neuron configured to read out a first value in accordance with an exemplary embodiment of the present disclosure.

7B depicts an example of a circuit of a one-time programmable memory circuit of a hardware neuron configured to read out a second value in accordance with an exemplary embodiment of the present disclosure.

8A depicts an exemplary one-time programming of a memory circuit of a storage bit using a first value in accordance with an exemplary embodiment of the present disclosure.

8B depicts an exemplary one-time programming of a memory circuit of a storage bit using a second value in accordance with an exemplary embodiment of the present disclosure.

Figure 9 depicts an exemplary configuration of a memory circuit of a hardware neuron according to an exemplary embodiment of the present disclosure.

Figure 10 depicts a flowchart of an exemplary method for operating hardware neurons in accordance with one aspect of the present disclosure.

Likewise, numerous embodiments are described and illustrated herein. The disclosure is neither limited to any single aspect or embodiment thereof, nor to any combination and/or permutation of these aspects and/or embodiments. Each aspect of the disclosure and/or embodiments thereof may be used alone or in combination with one or more other aspects of the disclosure and/or embodiments thereof. For the sake of brevity, many of these combinations and permutations are not discussed individually in this article.

As used herein, the terms “comprises,” “comprising,” or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but and may include other elements not expressly listed or inherent to such process, method, article, or equipment. The term "exemplary" is used in the sense of "example" rather than "ideal."

Detailed ways

Detailed illustrative aspects are disclosed herein. However, specific structural and functional details disclosed herein are representative only for the purpose of describing example embodiments of the present disclosure. The disclosure may be embodied in many alternative forms and should not be construed as limited to the embodiments set forth herein. Additionally, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the exemplary embodiments described herein.

When this specification refers to "one embodiment" or "an embodiment," it is intended that a particular feature, structure, characteristic, or function described in connection with the embodiment at issue is included in at least one contemplated embodiment of the present disclosure. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places in the specification do not constitute multiple references to a single embodiment of the disclosure.

As used herein, the singular forms "a", "an", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be noted that, in some alternative implementations, the features and/or steps described may occur out of the order depicted in the figures or discussed herein. For example, two steps or diagrams shown in succession may instead be performed substantially simultaneously, or sometimes in reverse order, depending on the functionality/actions involved. In some aspects, one or more of the described features or steps may be omitted entirely, or performed with intervening steps, depending on the function/action involved, without departing from the scope of the embodiments described herein.

Furthermore, the terms "first," "second," etc. used herein do not imply any order, quantity or importance, but are instead used to distinguish one element from another. Similarly, orientation-related terms, such as "top," "bottom," etc., are used with reference to the orientation of the structures illustrated in the figures being described. It should also be noted that all values disclosed herein may vary by ±10% from the disclosed value (unless a different variation is specified). In addition, all relative terms, such as "about," "substantially," "approximately," etc., are used to represent possible variations of ±10% (unless otherwise stated or designated another variation).

In one aspect, the present disclosure relates to techniques and implementations for programming storage devices, including, for example, non-volatile or "persistent" memory (e.g., Flash memory, MRAM) capable of retaining data when power is disabled. or ReRAM). Although the following description refers to MRAM or ReRAM memory device cells, the invention may be implemented in other memory devices, including but not limited to electrically erasable programmable read-only memory (EEPROM) and/or ferroelectric random access memory ( FRAM).

The present disclosure relates to systems and methods for storage bits in artificial neural networks that may address one or more of the above-mentioned problems. For example, according to certain embodiments, artificial neural network components (eg, associated with weight values, bias values, processing layers, etc.) may be stored using distributed magnetoresistive random access memory (MRAM) bits. In such edge distributed memory networks, one or more MRAM bits may be physically close to one or more hardware neurons or the hardware of an artificial neural network (e.g., within 500 micrometers (um) of each hardware neuron or within 500um of a functional hardware block within a hardware neuron) and can be used to store the artificial neural network components of that hardware neuron. One or more different MRAM bits may be physically proximate to one or more other hardware neurons of the same artificial neural network, and different MRAM bits may be used to store the artificial neural network component of that other hardware neuron.

As described elsewhere herein, an artificial neural network may include an input layer and an output layer. The input layer can receive one or more inputs into the artificial neural network. Input provided via the input layer may be applied to one or more hidden layers containing hardware neurons. One or more hidden layers can be trained based on supervised, semi-supervised or unsupervised machine learning. Each neuron can have multiple components (eg, weights, biases, layers, etc.) stored in memory. During the training process of training an artificial neural network, components of one or more hardware neurons can be accessed, modified, deleted, rewritten, added, etc. Therefore, a large amount of memory access may be required during the training of artificial neural networks. Additionally, during production use of the trained artificial neural network, components of the hardware neuron may be accessed and/or applied via corresponding memory accesses. Additionally, artificial neural networks can continue to be trained during the production process (e.g., based on feedback). Therefore, components of the Hardware Neuron may be modified, deleted, and/or added during the production process. In inference applications of artificial neural networks, multiple components of each neuron (e.g., weights or biases) may have to be stored in non-volatile memory. Conventionally, this is done by storing the weights or biases in flash memory. Data from external flash memory can be loaded into the ANN processor prior to inference applications and stored in locally available volatile storage elements such as SRAM, scan chains, or registers. In this conventional approach, moving data and storage elements may require additional power consumption.

As such, one or more of the above issues may be addressed by certain implementations described herein. For example, power consumption, computing resources, and/or time may be reduced based on the distributed storage (eg, MRAM) architecture disclosed herein. Continuing with the previous example, certain embodiments disclosed herein may reduce power consumption by providing on-chip access (e.g., instead of off-chip access) to artificial neural network components (e.g., weight values, bias values, processing layers, etc.) , computing resources and/or latency. Additionally, by having on-chip access, some implementations can reduce the amount of routing required to provide values from memory to processing circuitry, which can save chip space, reduce or eliminate circuitry in artificial neural networks, and so on.

Referring now to FIG. 1 , depicted is a functional diagram of an exemplary artificial neural network 100 in accordance with an exemplary embodiment of the present disclosure. As shown, artificial neural network 100 may include an input layer 102, a hidden layer 104, and an output layer 106. Input layer 102 can provide input values 108 to hidden layer 104, which can process input values 108. Hidden layer 104 may include one or more hardware neurons 110 (also referred to herein as neuron devices) for performing processing, and hidden layer 104 may provide processing results to output layer 106 (e.g., hardware neurons of output layer 106 112), used for output to the user, for further processing, etc.

As described in greater detail herein, the weight values and bias values may be stored in non-volatile memory and may be used during operation of the artificial neural network 100 . For example, a weight value may be associated with each arc (or synapse) between the input layer 102 and the hidden layer 104 and between the hidden layer 104 and the output layer 106 . Arcs are shown in Figure 1 as arrows between these layers. Additionally or alternatively, a bias value may be associated with each hardware neuron 110, 112 in the artificial neural network 100.

Although certain embodiments may be described herein in the context of artificial neural network 100, certain embodiments may be applicable to feedforward neural networks, radial basis function neural networks, Kohonen self-organizing neural networks, recurrent neural networks (RNN), Convolutional neural network (CNN), modular neural network (MNN), etc.

FIG. 2 depicts an example 200 of a first hardware neuron 110 of the artificial neural network 100 of FIG. 1 according to an exemplary embodiment of the present disclosure. For example, FIG. 2 depicts a functional diagram of hardware neuron 110 of artificial neural network 100 of FIG. 1; however, certain implementations may be equally applicable to hardware neuron 112.

As shown, the hardware neuron 110 may include a weighting circuit 114 that may be configured to perform an operation, such as a multiplication operation, on the input value 108 . For example, the multiplication operation may include multiplying the input value 108 received at the hardware neuron 110 by one or more weight values 122 associated with the hardware neuron 110 . Weight values 122 may be stored in storage circuitry 118 proximate hardware neurons 110 and/or weight operation circuitry 114 . Weight operation circuit 114 may read weight value 122 from storage circuit 118 and may multiply one or more input values 108 by weight value 122 . The weight operation circuit 114 may use a multiplier circuit to multiply the input value 108 by the weight value. As a specific example, the weight operation circuit 114 may multiply the input value 108a by the weight value 122a (eg, a ₁ *W ₁ ). In certain embodiments, weight values 122 may be updated based on a feedback loop, such as during training of artificial neural network 100 .

Hardware neuron 110 may also include bias operation circuitry 116 , which may be configured to perform an operation, such as an adder or summation operation, on the output of weight operation circuit 114 . For example, the bias operation circuit 116 may add one or more bias values 124 to the weighted value output from the weight operation circuit 114 . Bias values 124 may be stored in storage circuitry 118 proximate hardware neurons 110 and/or bias operation circuitry 116 . The deviation operation circuit 116 may read the deviation value 124 from the storage circuit 118 and may add the deviation value 124 to the weight value output from the weight operation circuit 114 . In some implementations, the deviation operation circuit 116 may add the deviation value 124 using a summation circuit. As a specific example, the weighting value output from the weighting operation circuit 114 (eg, the weighting value [a ₁ *W ₁ ] of the input value 108 a ) may be added to the deviation value 124 (e.g., the deviation operation circuit 116 may generate and ( a ₁ *W ₁ +b ₁ ) biased weighted value).

Storage circuitry 118 (eg, configured to store bits or configuration bits) may additionally be included in hardware neuron 110 . Storage circuitry 118 may include non-volatile memory, such as MRAM bits, that stores one or more weight values or bias values. For example, the storage circuits 118a and 118b can store the weight values 122a and 122b, and the weight operation circuits 114a and 114b can respectively read the weight values. As another example, storage circuit 118c may store offset value 124, which offset operation circuit 116 may read.

Storage circuitry 118 may store a single bit or may store multiple bits for different operating configurations. For example, storage circuit 118a may store a first weight value for a first operating condition, a second weight value for a second operating condition, and so on. As described in greater detail herein, memory circuitry 118 may include bridge elements (eg, MTJ bridges) and voltage amplifier circuits for each bit.

In this manner, hardware neurons 110 may be associated with multiple sets of memory circuits 118, each set corresponding to a different arithmetic circuit 114, 116. In addition, in this way, the storage circuit 118 can be located close to the corresponding arithmetic circuit 114, 116, which can reduce power consumption and/or delay in reading values from the storage circuit 118. Depending on the circuit layout of hardware neuron 110, certain implementations may include combined memory circuits 118 for weighting circuits 114a, 114b (e.g., memory circuits 118a, 118b may be combined into a set of memory circuits 118, where memory circuit 118c is A separate set of memory circuits 118); or the memory circuits 118a, 118c may be combined into a set of memory circuits 118 but store different types of values.

Memory circuitry 118 (eg, MRAM memory bits or configuration bits) may include one or more MTJs or other types of resistive elements. For example, and as described in greater detail herein, memory circuit 118 may include multiple MTJ bridge elements. The MTJ may have write and read capabilities using the product voltage drain supply (VDD) such as 0.8V, 1V, 1.2V, or 1.5V.

As further shown in FIG. 2 , the deviation operation circuit 116 may output the results of performing certain operations to the activation function circuit 120 , which may implement a ReLU activation function or a sigmoid activation function. Activation function circuit 120 may output values to hardware neurons 112 of output layer 106 . Hardware neuron 112 may include similar circuit configurations described for hardware neuron 110 . For example, different sets of arithmetic circuits of hardware neuron 112 may each be associated with a set of storage circuits 118 for storing values used in the operation of output layer 106 of hardware neuron 112 . The storage circuitry of hardware neuron 112 may be different from the storage circuitry 118 of hardware neuron 110 , for example, to facilitate positioning of storage circuitry 118 of hardware neuron 112 close to components of hardware neuron 112 .

FIG. 3 depicts an example 300 of the second hardware neuron 110 of the artificial neural network 100 of FIG. 1 according to an exemplary embodiment of the present disclosure. For example, FIG. 3 depicts a functional diagram of hardware neuron 110 of artificial neural network 100 of FIG. 1 (eg, FIG. 3 depicts an alternative configuration of hardware neuron 110 to the hardware neuron configuration depicted in FIG. 2).

As shown, the hardware neuron 110 of FIG. 3 may include weight operation circuits 114a, 114b, bias operation circuit 116, and activation function circuit 120 similar to the example 200 shown in FIG. 2. Hardware neuron 110 may also include memory circuitry 118 . However, rather than including multiple sets of memory circuits 118 for different arithmetic circuits 114, 116, example 300 may include one set of memory circuits 118 for storing weight values 122a, 122b and bias values 124. In example 300, storage circuitry 118 may include mini-arrays, and different hardware neurons 110 of artificial neural network 100 may include different mini-arrays. In some embodiments, artificial neural network 100 may include multiple memory circuit arrays 118 distributed over artificial neural network 100 (rather than the single array shown in Figure 3). For example, each hardware neuron 110 of the hidden layer 104 and/or each hardware neuron 112 of the output layer 106 may include an array similar to that shown as storage circuitry 118 in FIG. 3 .

4 depicts a configuration 400 of an exemplary storage circuit 118 of a hardware neuron in accordance with an exemplary embodiment of the present disclosure. For example, FIG. 4 depicts circuitry of a multiple-times programmable memory circuit 118 (eg, a storage or configuration bit) configured for sensing a first value or a second value, in accordance with an exemplary embodiment of the present disclosure. For example, memory circuit 118 may be a MRAM (eg, switching MRAM or spin transfer torque (STT) MRAM) or ReRAM that may be reprogrammed multiple times to represent different values. The circuitry of memory circuit 118 shown in FIG. 4 can read out a first value (eg, a 0 value in a binary system of 0s and 1s) or a second value (eg, a 1 value in a binary system of 0s and 1s).

As shown, memory circuit 118 may include an MTJ bridge 402, a voltage amplifier 404, and an inverter (not shown in Figure 4). MTJ bridge 402 may include one or more resistive elements 408 (eg, resistive elements 408a, 408b, 408c, and 408d). Although FIG. 4 illustrates the MTJ bridge 402 as including four resistive elements 408, certain embodiments may include any number of multiple resistive elements 408 greater than four (eg, 5, 6, 7, 8 resistive elements). Resistive element 408 may include an MTJ or another type of electrical component capable of providing resistance to the flow of electrical current. For example, resistive element 408 may have multiple resistance states (eg, a low resistance state (parallel) Rp and a high resistance state (antiparallel) Rap).

MTJ bridge 402 may also include one or more electrodes 412 (eg, electrodes 412a, 412b, 412c, and 412d) to electrically connect different resistive elements 408 in series or parallel. For example, the MTJ bridge 402 may include four resistive elements, wherein two first resistive elements are electrically connected in series and two second resistive elements are electrically connected in series and wherein the first resistive element and the second resistive element are electrically connected in parallel. As a specific example, resistive elements 408a, 408b (forming the first set of resistive elements 408) may be electrically connected in series via electrode 412a, resistive elements 408c, 408d (forming the second set of resistive elements 408) may be electrically connected in series via electrode 412b, and One set and a second set of resistive elements may be electrically connected in parallel via electrodes 412c, 412d.

As further shown in Figure 2, storage circuitry 118 may include one or more electrical connections 410 (eg, electrical connections 410a, 410b, 410c, 410d, and 410e). Electrical connection 410a may electrically connect electrode 412a to a voltage source (not shown in Figure 4) and electrical connection 410b may electrically connect electrode 412b to the voltage source. Electrical connection 410c may electrically connect electrode 412c to the input of voltage amplifier 404 and electrical connection 410d may electrically connect electrode 412d to the input of voltage amplifier 404. Electrical connection 410e may electrically connect the output of the voltage amplifier to the inverter (not shown in Figure 4). Depending on whether the inverter's gate is open or closed, the inverter may be in different states. Based on the voltage applied to the MTJ bridge 402, the inverter may be in a first state (eg, a 1 state) indicating a first value (eg, a 1 value).

As mentioned above, the resistive element 408 may have two resistance states (eg, a high resistance state Rap and a low resistance state Rp). For the first state of the inverter, resistive elements 408a, 408d may be in a high resistance state and resistive elements 408b, 408c may be in a low resistance state. For the second state of the inverter, resistive elements 408a, 408d may be in a low resistance state and resistive elements 408b, 408c may be in a high resistance state.

In some embodiments, the MTJ bridge 402 of the memory circuit 118 shown in Figure 4 can store one bit, and the memory circuit 118 can be configured with multiple instances of the MTJ bridge 402 shown in Figure 4 for multiple bits. . As described elsewhere herein, the MTJ bridge 402 may be read, multiple times programmed (MTP), and/or one time programmed (OTP).

FIG. 5 depicts various bridge element configurations 500 for memory circuitry 118 of hardware neuron 110 in accordance with an exemplary embodiment of the present disclosure. For example, different bridge element configurations 402a, 402b, 402c, 402d, and 402e may provide storage of different values. In configurations in which memory circuit 118 includes multiple bits (eg, multiple instances of MTJ bridge 402 ), memory circuit 118 may include multiple bridge element configurations 500 that may each be configured identically based on configuration 500 or different values. In other configurations in which memory circuit 118 includes a single bit (eg, a single instance of MTJ bridge 402), the memory bit may be programmed multiple times into configuration 500 for storing different values.

Bridge element configuration 500 may store different values based on different resistance (Rp and Rap) configurations of resistive element 408 . For example, the resistance values of one or more resistors and/or active resistors (eg, four MTJs as resistive elements 408) may be configured to output various combinations of bit values. A single MTJ bridge 402 may output two or more states based on its configured (eg, stored) resistance value. Voltage amplifiers with multiple threshold levels can be used to output multiple states (eg, more than two outputs) from the same MTJ bridge element 402 .

Therefore, one or more configuration bits may use the MTJ bridge 402 to store larger amounts or more complex data using various resistor configuration bits. For example, artificial neural network 100 may have to use multiple bits to store weight values and/or bias values. One or more configurations of resistive element 408 may be used (eg, by modifying the resistance value) to store weight values and/or bias values using multiple bits. In this manner, bridge element 402 may be used to store one or more bits of data based on different configurations 500 . In some implementations, configuration 500 may include one or more sensing circuits.

As such, while artificial neural network 100 may have to use large amounts of storage space (eg, gigabit or more) across artificial neural network 100, certain embodiments described herein may be used in close proximity to hardware neurons 110, 112 ( or arithmetic circuitry of hardware neurons 110, 112) with a small storage space (eg, 1 to 8 MRAM bits). This may facilitate sizing a memory circuit (eg, memory circuit 118) based on the operation of hardware neurons 110, 112 rather than on the operation of the entire artificial neural network 100. This can save chip space, allow hardware neurons 110, 112 to access stored information faster and at lower power, etc.

FIG. 6A depicts an example 600 of a multiple-time programmable storage circuit 118 configured to write a first value to a hardware neuron (eg, hardware neuron 110 or hardware neuron 112 ) in accordance with an exemplary embodiment of the present disclosure. . Example 600 may include an MTJ bridge 402, voltage amplifier 404, inverter, resistive element 408, electrical connections 410, and electrodes 412 configured in a manner similar to the configuration 400 shown in FIG. 4 (some of which are not shown for illustration purposes). shown in Figure 6).

Based on positive Vdd applied to electrode 412c (eg, first bottom electrode) and ground voltage (GND) applied to electrode 412d (eg, second bottom electrode), the inverter (not shown in FIG. 6A ) may be in the indicated A first state (eg, 0 state) of a first value (eg, 0 value). In this state, upon application of Vdd and GND, current can flow from electrode 412c up through resistive element 408a and down through resistive element 408c, through electrodes 412a, 412b (eg, top electrode), and then down through resistive element 408b and upward through resistive element 408d to electrode 412d. The positive Vdd applied to electrode 412c may be higher than the switching voltage of the resistive element and lower than the breakdown voltage of the resistive element.

Turning to FIG. 6B , depicted is an example 600 of a circuit of the multi-times programmable memory circuit 118 configured to write a second value in accordance with an exemplary embodiment of the present disclosure. Example 600 may include an MTJ bridge 402, voltage amplifier 404, inverter, resistive element 408, electrical connections 410, and electrodes 412 configured in a manner similar to example 600 shown in FIG. 6B (some of which are not shown for illustration purposes). shown in Figure 6B).

Based on the positive Vdd applied to electrode 412d (eg, the second bottom electrode) and the GND voltage applied to electrode 412c (eg, the first bottom electrode), the inverter (not shown in FIG. 6B ) may be in a state indicating a second value (e.g., 1 value). In this state, upon application of Vdd and GND, current can flow from electrode 412d up through resistive element 408b and down through resistive element 408d, through electrodes 412a, 412b (eg, top electrode), and then down through resistive element 408a and upward through resistive element 408c to electrode 412c.

7A depicts an example 700 of circuitry for a one-time programmable memory circuit 118 of a hardware neuron configured to sense a first value in accordance with an exemplary embodiment of the present disclosure. For example, memory circuit 118 may not be reprogrammable to another value. Example 700 may include an MTJ bridge 402, voltage amplifier 404, inverter 406, resistive element 408, electrical connection 410, and electrode 412 configured in a manner similar to configuration 400 shown in FIG. 4 . However, instead of having the resistive elements 408b, 408c in a low or high resistance state, the resistive elements 408b, 408c can be short-circuited (identified by "Short Circuit" in Figure 7A). Shorting these resistive elements may permanently place inverter 406 in a first state (eg, 1 state) indicating a first value (eg, 1 value).

Turning to FIG. 7B , depicted is a one-time programmable memory circuit 118 of a hardware neuron (eg, hardware neuron 110 or hardware neuron 112 ) configured to sense a second value in accordance with an exemplary embodiment of the present disclosure. Example of circuit 700. For example, memory circuit 118 may not be reprogrammable to another value. Example 700 may include an MTJ bridge 402, voltage amplifier 404, inverter 406, resistive element 408, electrical connection 410, and electrode 412 configured in a similar manner to example 400 shown in FIG. 4 . However, rather than leaving resistive elements 408a and 408d in a low or high resistance state, resistive elements 408a and 408d can be shorted. Shorting these resistive elements 408 may permanently leave the inverter 406 in a second state (eg, a 0 state) indicating a second value (eg, a 0 value).

8A depicts an exemplary one-time programming 800 of a storage circuit 118 using a first value in accordance with an exemplary embodiment of the present disclosure. The circuit may include an MTJ bridge 402, voltage amplifier 404, inverter, resistive element 408, electrical connections 410, and electrodes 412 similar to those described elsewhere herein (some of which are not shown in Figure 8A for illustration purposes) . Resistive elements 408a, 408b may form a first set of resistive elements 408 and resistive elements 408c, 408d may form a second set of resistive elements 408.

The programming may include two steps 802, 804 for configuring the circuit in a manner similar to that described above in conjunction with example 700 of Figure 7. The first step 802 may include applying various voltages across the resistive element 408 (eg, simultaneously or at different times). For example, a relatively high (compared to Vdd) programming voltage (Vprog) 806 may be applied across resistive element 408b (one of the first set of resistive elements 408) to short-circuit resistive element 408b. In this manner, a positive voltage may be applied from electrode 412d across resistive element 408b to electrode 412a to program memory circuit 118 with the first value.

The second step 804 may include applying various voltages across the resistive element 408 (eg, simultaneously or at different times). For example, a relatively high (compared to Vdd) programming voltage (Vprog) 814 may be applied across resistive element 408c (one of the second set of resistive elements 408) to short circuit resistive element 408c. In this manner, a positive voltage may be applied from electrode 412b across resistive element 408c to electrode 412c to program memory circuit 118 with the first value.

Turning to FIG. 8B , an exemplary one-time programming 800 of a storage circuit 118 using a second value is depicted in accordance with an exemplary embodiment of the present disclosure. The circuit may include an MTJ bridge 402, voltage amplifier 404, inverter, resistive element 408, electrical connections 410, and electrodes 412 similar to those described elsewhere herein (some of which are not shown in Figure 8B for illustration purposes) . Resistive elements 408a, 408b may form a first set of resistive elements 408 and resistive elements 408c, 408d may form a second set of resistive elements 408.

The programming may include two steps 816, 818 for configuring the circuit in a manner similar to that described above in conjunction with example 700 of Figure 7B. The first step 816 may include applying various voltages across the resistive element 408 (eg, simultaneously or at different times). For example, a relatively high Vprog 820 may be applied across resistive element 408a (one of the first set of resistive elements 408) to short circuit resistive element 408a. In this manner, a positive voltage may be applied from electrode 412c across resistive element 408a to electrode 412a to program memory circuit 118 with the second value.

The second step 818 may include applying various voltages across the resistive element 408 (eg, simultaneously or at different times). For example, a relatively high Vprog 826 may be applied across resistive element 408d (one of the second set of resistive elements 408) to short circuit resistive element 408d. In this manner, a positive voltage may be applied from electrode 412b across resistive element 408d to electrode 412d to program memory circuit 118 with the second value.

9 depicts an exemplary configuration 900 of storage circuitry 118 of a hardware neuron (eg, hardware neuron 110 or hardware neuron 112) according to an exemplary embodiment of the present disclosure. For example, Figure 9 shows an alternative configuration to the memory circuit 118 shown in Figures 4-8b. Example configuration 900 may include various read circuit sets 902 . For example, memory circuit 118 may include read circuit 902a including two transistors, read circuit 902b including one transistor, and read circuit 902c including one transistor. Read circuit 902a may be electrically connected to cross-coupled inverter circuit 904 via a voltage source (Vsup) connection. Cross-coupled inverter circuit 904 may include four transistors and may include output circuits 906a (labeled "out" in Figure 9) and 906b (labeled "out_b" in Figure 9). Read circuit 902b may be associated with storage bit circuit 908a and may read the value stored in storage bit circuit 908a. Read circuit 902c may be associated with storage bit circuit 908b and may read the value stored in storage bit circuit 908b.

Cross-coupled inverter circuit 904 may generate outputs out and out_b (out_b may be the opposite polarity signal of the out output) indicating the state of the MRAM storage bits. During a read operation, read circuit 902a may transition from VDD to ground (Gnd), causing Vsup to transition from Gnd to VDD and causing out/out_b to no longer be pulled down to Gnd. The current difference between memory bit circuits 908a and 908b may cause the out and out_b circuits to provide full swing (Gnd or VDD) outputs. The MTJ state in memory bit circuits 908a and 908b can create a current difference. Memory bit circuit 908a or 908b may be implemented with a single MTJ or a series of two or more MTJs to reduce MTJ variation. Alternative configurations to the embodiment shown in Figure 9 are possible. For example, the MTJ bridge may be connected to the cross-coupled inverter circuit 904 in any other configuration to respond to voltage or current differences.

The series connection of the MTJs in the storage bit circuits 908a and 908b can help ensure that the read current through any MTJ is minimized to avoid any read interruption of the stored MTJ state. During write operations, other p-type metal oxide semiconductor (PMOS) and n-type metal oxide semiconductor (NMOS) transistors (not shown in Figure 9) can be connected to the MTJ bridge to write to one or more MTJs at a time ( For example, writing two or multiples of two MTJs at a time). Therefore, the write current can pass through at least two MTJs in series in a manner similar to that shown in Figures 6a-7b. In this way, certain implementations may provide no quiescent current consumption after reading a memory bit. An alternative implementation (not shown here) may be used to perform the same functionality as described above, with a cross-coupled inverter circuit similar to the cross-coupled inverter circuit shown in FIG. 9 . For example, MTJ bridges 908a, 908b may be located between the Vsup node and the cross-coupled inverter circuit 904. Additional NMOS transistors acting as follower circuits can control the voltage applied across the MTJ bridges 908a, 908b.

Figure 10 depicts a flowchart of an exemplary method 1000 for operating hardware neurons 110 in accordance with one aspect of the present disclosure. For example, method 1000 may use hardware neurons 110 in conjunction with the operation of artificial neural network 100 .

In step 1002, method 1000 may include receiving a value at a weighting circuit of a device via an input circuit of the device. For example, hardware neuron 110 may receive value 108 via an input circuit of input layer 102 at weight operation circuit 114 of hardware neuron 110 . In the context of Figures 2 and 3 above, hardware neuron 110 may receive values 108a and 108b, respectively, at weight operation circuit 114. Hardware neurons 110 may receive the values as part of the training process of artificial neural network 100 and may receive various input values 108 throughout the training process.

In step 1004, method 1000 may include applying, at a weighting circuit, a weight value from a memory circuit of the device to the value to form a weighted value. For example, hardware neuron 110 may apply weight values 122 from storage circuit 118 of hardware neuron 110 at weight operation circuit 114 to form weighted values. The application may include hardware neuron 110 multiplying value 108 by weight value 122 using weighting circuit 114 . For example, and as described elsewhere herein, hardware neuron 110 may multiply value a ₁ by weight value W ₁ to form product a ₁ W ₁ . In the context of FIG. 2 above, hardware neuron 110 may apply weight value 122a from storage circuit 118a to input value 108a at weight operation circuit 114a and may apply weight value 122b from storage circuit 118b at weight operation circuit 114b. Applies to input value 108b. In the context of FIG. 3 above, hardware neuron 110 may apply weight value 122a from storage circuit 118 to input value 108a at weight operation circuit 114a and may apply weight value 122b from storage circuit 118 at weight operation circuit 114b. Applies to input value 108b.

In some embodiments, in conjunction with applying the weight value 122 to the input value 108, the weight operation circuit 114 may read the weight value 122 from the storage circuit 118, may receive a transmission of the weight value 122 from the storage circuit 118, and so on.

Method 1000 may include providing, at step 1006, a weighted value to a bias operation circuit of the device. For example, hardware neuron 110 may provide weighted values to bias operation circuitry 116 of hardware neuron 110 . As a specific example, hardware neuron 110 may provide weight value a ₁ W ₁ from weight operation circuit 114 to bias operation circuit 116 after applying weight value 122 to input value 108 . In the context of Figures 2 and 3, the hardware neuron 110 may provide the weighting values calculated at the weighting circuits 114a, 114b to the biasing circuit 116.

At step 1008, method 1000 may include applying the bias value from the storage circuit to the weighted value at the bias operation circuit to form the biased weighted value. For example, hardware neuron 110 may apply bias value 124 from storage circuit 118 to a weighted value at bias operation circuit 116 to form a biased weighted value. In the context of Figure 2, the hardware neuron 110 may apply the bias value 124 from the storage circuit 118c at the bias operation circuit 116 to the weighted values received from the weight operation circuits 114a, 114b. As a specific example, bias operation circuit 116 may add bias value 124 to the weighted value from weight operation circuit 114 (e.g., bias operation circuit 116 may generate a biased weighted value sum (a ₁ *W ₁ +b ₁ )) . In the context of Figure 3, the hardware neuron 110 may apply the bias value 124 from the storage circuit 118 at the bias operation circuit 116 to the weighted values received from the weight operation circuits 114a, 114b.

Method 1000 may include providing, at 1010, a biased weight value to an activation function circuit of the device. For example, hardware neuron 110 may provide the biased weighted value from bias operation circuit 116 to activation function circuit 120 after applying bias value 124 to the weighted value from weight operation circuit 114 . In the context of FIGS. 2 and 3 , hardware neuron 110 may provide sum (a ₁ *W ₁ +b ₁ ) and sum (a ₂ *W ₂ +b ₂ ) from bias operation circuit 116 to activation function circuit 120 .

Method 1000 may include providing, at 1012, an output from an activation function circuit to an output circuit of the device. For example, hardware neuron 110 may provide the output from activation function circuit 120 to an output circuit of hardware neuron 110 and then to hardware neuron 112 of output layer 106 .

Certain embodiments described herein may include additional or alternative aspects. As an example aspect, storage circuit 118 may be reprogrammed with updated weight values 122 or bias values 124 and certain operations of method 1000 may be re-performed based on the updated values.

Certain implementations described herein are capable of tolerating high error rates in artificial neural network 100 applications. In this way, acceptable and unacceptable error rates can be identified based on error tolerance, and in some embodiments, error correcting code (ECC) can be omitted based on high error tolerance, or error correcting code (ECC) can be implemented such that If high fault tolerance is met, ECC is activated. Therefore, the memory bits can implement ECC bits and ECC correction according to the required bit error rate. This can save resources and/or chip space associated with implementing ECC or implementing ECC with a lower error rate threshold.

As such, certain embodiments described herein may provide on-chip storage of certain values using circuitry proximate to the circuitry in which the values are to be used. Using such on-chip storage, the time and computational resource costs (e.g., power consumption) of retrieving, storing, and/or updating such values can be reduced. Certain implementations disclosed herein, such as MTJ-based circuit configurations, may enable multiple bits of storage per MTJ bridge. Additionally or alternatively, on-chip access to memory may reduce or eliminate the risk of connection loss that would otherwise be associated with external memory access. Additionally or alternatively, certain embodiments may enhance the security of weight values and/or bias values of a trained network, such as in inference applications. Additionally or alternatively, certain embodiments may enable writing of memory bits in MTP mode, such as in training applications, which may save power and/or reduce latency compared to using off-chip non-volatile memory . For example, in learning applications, the weight values 122 and bias values 124 may need to be continuously adjusted, thus resulting in frequent memory accesses; and locating the multi-time programmable memory circuit 118 near the operating circuits 114, 116 can reduce training time and training. associated power consumption.

In one embodiment, an apparatus may include: an input circuit; a weight operation circuit electrically connected to the input circuit; a deviation operation circuit electrically connected to the weight operation circuit; a storage circuit, the storage circuit is electrically connected to the weight operation circuit and the deviation operation circuit; and an activation function circuit, the activation function circuit is electrically connected to the deviation operation circuit, wherein at least the weight operation circuit, the deviation operation circuit The deviation operation circuit and the storage circuit are located on the same chip.

Various implementations of the apparatus may include: wherein the weighting circuit includes a first weighting circuit and a second weighting circuit, and wherein the storage circuit includes a first weighting circuit electrically connected to the first weighting circuit. a storage circuit, a second storage circuit electrically connected to the second weight operation circuit, and a third storage circuit electrically connected to the deviation operation circuit; wherein the weight operation circuit includes a first weight operation circuit and a second weight operation circuit circuit, and wherein the storage circuit is electrically connected to the first weight operation circuit, the second weight operation circuit and the deviation operation circuit; wherein the storage circuit includes one or more storage bits; wherein the one or a plurality of storage bits each including one or more resistive elements and a voltage amplifier; wherein the one or more resistive elements include at least four resistive elements, wherein at least two first resistive elements are electrically connected in series and at least two second resistive elements The resistive elements are electrically connected in series, wherein the at least two first resistive elements are electrically connected in parallel with the at least two second resistive elements, and wherein the input terminal of the voltage amplifier is electrically connected to the at least two first resistors. a first electrode between the elements and connected to a second electrode between the at least two second resistive elements; wherein each of the one or more resistive elements includes a magnetic tunnel junction (MTJ); wherein the The one or more storage bits are included in a single bit array; wherein the device includes hardware neurons in an artificial neural network; the device further includes an output circuit electrically connected to the activation function circuit; wherein the one Each of the or plurality of storage bits includes a first set of resistive elements and a second set of resistive elements, a first read circuit electrically connected to the first set of resistive elements and a first read circuit electrically connected to the second set of resistive elements. a second read circuit of the element, a cross-coupled inverter circuit electrically connected to the first read circuit and the second read circuit, and a third read circuit, the The third read circuit is electrically connected to the cross-coupled inverter circuit.

In another embodiment, a neuron device for an artificial neural network may include: an input circuit; a weight operation circuit electrically connected to the input circuit; and a deviation operation circuit, the deviation operation circuit Electrically connected to the weight operation circuit; a storage circuit, the storage circuit is electrically connected to the weight operation circuit and the deviation operation circuit; and an activation function circuit, the activation function circuit is electrically connected to the deviation operation circuit, Wherein at least the weight calculation circuit, the deviation calculation circuit and the storage circuit are located on the same chip.

Various embodiments of the neuron device may include: wherein the weighting circuit includes a first weighting circuit and a second weighting circuit, and wherein the storage circuit includes a memory circuit electrically connected to the first weighting circuit. A first storage circuit, a second storage circuit electrically connected to the second weight operation circuit, and a third storage circuit electrically connected to the deviation operation circuit; wherein the weight operation circuit includes a first weight operation circuit and a second a weight operation circuit, and wherein the storage circuit is electrically connected to the first weight operation circuit, the second weight operation circuit and the deviation operation circuit; wherein the storage circuit includes one or more storage bits, wherein the Each of the one or more storage bits includes one or more resistive elements and a voltage amplifier; wherein the one or more resistive elements include at least four resistive elements, wherein at least two first resistive elements are electrically connected in series and at least two second resistive elements are electrically connected in series, wherein the at least two first resistive elements are electrically connected in parallel with the at least two second resistive elements, and wherein the input terminal of the voltage amplifier is electrically connected to the a first electrode between at least two first resistive elements and connected to a second electrode between said at least two second resistive elements; wherein said one or more storage bits are included in a single bit array; said The neuron device further includes an output circuit electrically connected to the activation function circuit; wherein each of the one or more storage bits includes: a first set of resistive elements and a second set of resistive elements electrically connected to the a first reading circuit of the first set of resistive elements and a second reading circuit electrically connected to the second set of resistive elements, a cross-coupled inverter circuit electrically connected to the A reading circuit and the second reading circuit and a third reading circuit are electrically connected to the cross-coupled inverter circuit.

In yet another embodiment, a method of operating a device of an artificial neural network may include: receiving a value at a weight operation circuit of the device via an input circuit of the device; a weighted value of a storage circuit of the device is applied to the value to form a weighted value; the weighted value is provided to a deviation operation circuit of the device; and the deviation value from the storage circuit is applied to the deviation operation circuit at the deviation operation circuit the weighting value to form a biased weighting value; and providing the biased weighting value to an activation function circuit of the device, wherein at least the weight operation circuit, the bias operation circuit and the storage circuit are located in the same on the chip.

Although the principles of the disclosure are described herein with reference to illustrative examples of specific applications, it is to be understood that the disclosure is not limited thereto. For example, instead of MTJ-based bit cells, another memory bit such as resistive RAM or ferroelectric RAM bit technology may be used to design the antifuse circuit of the present disclosure. Another memory bit may have a programmed state and at least one unprogrammed state. The at least one unprogrammed state may also include a plurality of unprogrammed states, such as a low unprogrammed state, a high unprogrammed state, and one or more intermediate unprogrammed states. Those of ordinary skill in the art and having access to the teachings provided herein will recognize additional modifications, applications, implementations, and substitutions of equivalents that fall within the scope of the features described herein. Accordingly, the claimed features should not be regarded as limited by the foregoing description.

The foregoing description of the present invention has been presented for the purposes of clarity and understanding. It is not intended to limit the invention to the precise form disclosed. Various modifications may be made within the scope and equivalent scope of the present application.

Claims (20)

1. An apparatus, the apparatus comprising:

an input circuit;

a weight operation circuit electrically connected to the input circuit;

a bias operation circuit electrically connected to the weight operation circuit;

a storage circuit electrically connected to the weight operation circuit and the deviation operation circuit; and

an activation function circuit electrically connected to the bias operation circuit,

wherein at least the weight operation circuit, the deviation operation circuit and the storage circuit are located on the same chip.

2. The apparatus of claim 1, wherein the weight operation circuit comprises a first weight operation circuit and a second weight operation circuit, and

wherein the memory circuit includes a first memory circuit electrically connected to the first weight operation circuit, a second memory circuit electrically connected to the second weight operation circuit, and a third memory circuit electrically connected to the deviation operation circuit.

3. The apparatus of claim 1, wherein the weight operation circuit comprises a first weight operation circuit and a second weight operation circuit, and

wherein the storage circuit is electrically connected to the first weight operation circuit, the second weight operation circuit, and the deviation operation circuit.

4. The apparatus of claim 1, wherein the memory circuit comprises one or more memory bits.

5. The apparatus of claim 4, wherein the one or more memory bits each comprise one or more resistive elements and a voltage amplifier.

6. The apparatus of claim 5, wherein the one or more resistive elements comprise at least four resistive elements, wherein at least two first resistive elements are electrically connected in series and at least two second resistive elements are electrically connected in series, wherein the at least two first resistive elements are electrically connected in parallel with the at least two second resistive elements, and

wherein the input of the voltage amplifier is electrically connected to a first electrode between the at least two first resistive elements and to a second electrode between the at least two second resistive elements.

7. The device of claim 5, wherein each of the one or more resistive elements comprises a Magnetic Tunnel Junction (MTJ).

8. The apparatus of claim 4, wherein the one or more storage bits are included in a single bit array.

9. The apparatus of claim 1, wherein the apparatus comprises a hardware neuron in an artificial neural network.

10. The apparatus of claim 1, further comprising an output circuit electrically connected to the activation function circuit.

11. The device of claim 4, wherein each of the one or more storage bits comprises:

a first set of resistive elements and a second set of resistive elements,

a first read circuit electrically connected to the first set of resistive elements and a second read circuit electrically connected to the second set of resistive elements,

a cross-coupled inverter circuit electrically connected to the first and second read circuits, and

a third read circuit electrically connected to the cross-coupled inverter circuit.

12. A neuron device for an artificial neural network, the neuron device comprising:

an input circuit;

a weight operation circuit electrically connected to the input circuit;

a bias operation circuit electrically connected to the weight operation circuit;

A storage circuit electrically connected to the weight operation circuit and the deviation operation circuit; and

an activation function circuit electrically connected to the bias operation circuit,

wherein at least the weight operation circuit, the deviation operation circuit and the storage circuit are located on the same chip.

13. The neuron device according to claim 12, wherein the weight operation circuit includes a first weight operation circuit and a second weight operation circuit, and

wherein the memory circuit includes a first memory circuit electrically connected to the first weight operation circuit, a second memory circuit electrically connected to the second weight operation circuit, and a third memory circuit electrically connected to the deviation operation circuit.

14. The neuron device according to claim 12, wherein the weight operation circuit includes a first weight operation circuit and a second weight operation circuit, and

wherein the storage circuit is electrically connected to the first weight operation circuit, the second weight operation circuit, and the deviation operation circuit.

15. The neuron device of claim 12, wherein the storage circuit comprises one or more storage bits, wherein each of the one or more storage bits comprises one or more resistive elements and a voltage amplifier.

16. The neuron device according to claim 15, wherein the one or more resistive elements comprise at least four resistive elements, wherein at least two first resistive elements are electrically connected in series and at least two second resistive elements are electrically connected in series, wherein the at least two first resistive elements are electrically connected in parallel with the at least two second resistive elements, and

wherein the input of the voltage amplifier is electrically connected to a first electrode between the at least two first resistive elements and to a second electrode between the at least two second resistive elements.

17. The neuron device of claim 15, wherein the one or more storage bits are included in a single bit array.

18. The neuron device of claim 12 further comprising an output circuit electrically connected to the activation function circuit.

19. The neuron device of claim 15, wherein each of the one or more storage bits comprises:

a first set of resistive elements and a second set of resistive elements,

a first read circuit electrically connected to the first set of resistive elements and a second read circuit electrically connected to the second set of resistive elements,

A cross-coupled inverter circuit electrically connected to the first and second read circuits, and

a third read circuit electrically connected to the cross-coupled inverter circuit.

20. A method of operating a device of an artificial neural network, the method comprising:

receiving a value at a weight operation circuit of the device via an input circuit of the device;

applying weight values from a storage circuit of the device to the values at the weight operation circuit to form weighted values;

providing the weighted value to a bias operation circuit of the device;

applying, at the bias operation circuit, bias values from the storage circuit to the weighting values to form biased weighting values; and

providing the biased weighting value to an activation function circuit of the device,

wherein at least the weight operation circuit, the deviation operation circuit and the storage circuit are located on the same chip.